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1  Overview:  Introduction 


IP  network  technology  cannot  continue  on  an  ever-increasing  course  o  technological  complexity 
and  yet  require  the  kind  of  human  intervention  that  is  necessary  today  for  network  management. 
Networks  must  be  self-managing.  This  can  only  be  done  by  a  system  of  measurement  that  copes 
with  the  dynamics  of  packet  movement.  This  system  must  process  the  packet-level 
measurements  into  variables  that  characterize  network  behaviors,  which  then  form  the  basis  for 
control  algorithms  that  react  to  the  variables.  Such  packet-level  measurements  can  lead  to 
characterization  and  control  at  the  application  layer,  at  the  transport  layer,  at  the  network  layer 
(including  overlay  networking),  and  in  some  cases  at  the  link  layer.  This  research  will  use  tools 
of  statistics,  data  mining  and  machine  learning  to  explore  packet-level  measurement  for  self¬ 
management. 

The  current  dominant  measurement  framework  on  IP  networks,  SNMP,  is  not  adequate  to  the 
task  because  it  does  not  allow  the  determination  of  network  time  dynamics.  We  studied  data  that 
allows  the  determination  of  time  dynamics — time-stamping  the  arrivals  of  packets  at  network 
nodes  and  recording  header  contents.  The  research  used  methods  of  statistics,  data  mining,  and 
machine  learning  to: 

(1)  determine  network  variables  derivable  from  the  measurements  that  characterize 
network  behavior; 

(2)  develop  models  of  the  critical  network  variables  that  characterize  performance, 
usage,  security,  and  early  onset  of  problems;  and 

(3)  develop  automated  control  methods  based  on  the  variables.  In  addition,  certain 
models  that  we  developed  can  provide  traffic  generation  in  network  simulation  systems 
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that  can  be  used  for  further  study  of  control. 

Our  project  involved  the  study  of  previously  collected  packet  header  data  on  IP  network  links,  as  well  as 
providing  for  collection  of  new  data. 

The  statistics,  data  mining,  and  machine  learning  was  carried  out  using  the  S-Net  software  system  for  IP 
network  packet-level  data.  S-Net,  the  most  comprehensive  environment  for  packet  header  study  available 
today,  was  developed  with  support,  in  paid,  by  DARPA  under  federal  contract  No.  F30602-00-C-0034. 

This  report  describes  the  work  done  under  the  above  federal  contract  as  well  as  previous  work  done  under 
federal  contract  No.  F30602-02-2-C0093  awarded  to  Bell  Labs  because  the  work  under  the  new  contract 
was  a  continuation  of  the  work  under  the  previous  contract.  In  a  later  section  of  this  final  report  the  paid  of 
the  work  done  under  the  new  contract  is  described. 


2  Overview:  Main  Elements  of  the  Approach 


Assembled  a  database  of  packet  header  data.  Partnered  with  network  operators  who  were  a  source  not  only 
of  data,  but  provided  knowledge  of  the  critical  network  operations  issues.  Partnered  with  network  equip¬ 
ment  manufacturers  to  determine  feasiblity  of  different  packet-level  measurement  frameworks.  Statistical 
analysis,  data  mining,  and  machine  learning  used  to  study  network  behavior  using  the  S-Net  system  for 
packet-header  analysis  and  visualization.  Developed  models  for  the  measurements.  Developed  algorithms 
for  self-management  based  on  the  measurement  scenario,  models,  and  the  learned  network  behavior. 


3  Overview:  Deliverables 


1.  Management  and  administration:  As  paid  of  the  conventional  DARPA  monthly  reporting  process, 
project  management  deliverables  included  the  status  of  the  effort,  progress  toward  accomplishment  of  con¬ 
tract  requirements,  and  monitoring  funding  use  rates  (spreadsheet  format  for  financial  reports  with  actual 
monthly  and  cumulative  expenditures  compared  with  the  programmed  budget).  We  also  made  oral  presen¬ 
tations  at  two  DARPA  PI  meetings. 

2.  Research:  Results  on  measurement  frameworks,  network  characterization,  models,  variables  that  char¬ 
acterize  network  behavior,  and  self-managing  control  methods  based  on  the  variables,  conveyed  through 
viewgraphs  and  written  documents. 


4  Overview:  Statement  of  Work 


We  collaborated  and  cooperated  with  the  other  program  contractors  on  the  planning,  design,  implementation 
and  demonstration  of  new  technologies  described  within  this  proposal. 

Over  the  course  of  execution  of  the  contract  resulting  from  this  proposal,  we  maintained  effective  com¬ 
munications  and  active  exchange  with  all  other  traffic  modeling  program  researchers  in  order  to  facili¬ 
tate  free-flowing  information  transfer  and  integration  with  other  investigators.  We  actively  exchanged  re¬ 
search  results  with  the  larger  research  community  through  journal  publications  and  oral  presentations  at 
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national/international  seminars  and  meetings. 


We  documented  technical  work  accomplished  and  information  gained  during  performance  of  this  acquisi¬ 
tion  in  the  form  of  monthly  status  and  special  technical  reports  and  this  final  report.  These  reports  included 
significant  observations,  problems,  positive  results,  negative  results,  and  design  criteria.  These  reports  doc¬ 
umented  procedures  followed,  processes  developed,  lessons  learned,  and  other  useful  information.  The 
reports  documented  the  details  of  all  technical  work  to  permit  full  understanding  of  the  techniques  and 
procedures  used  in  evolving  technology  or  processes  developed. 

Task  1:  Packet-Level  Measurement 

We  assembled  a  database  of  packet-level  measurements  consisting  of  packet  header  fields  and  time-stamps. 
The  measurements  were  passive.  We  obtained  existing  measurements  and  deployed  devices  to  collect  more 
data. 

We  studied  best-effort  traffic  streams,  the  dominant  traffic  type  on  IP  networks  today,  as  well  as  voice  over 
IP  traffic.  Increasingly,  today’s  voice  traffic  is  moving  from  the  circuit-switched  telephone  networks  to  IP 
networks.  Many  service  providers  now  have  architectures  in  which  calls  start  and  end  on  the  circuit-switched 
networks,  but  are  carried  as  IP  network  packet  traffic  over  the  IP  network  core.  Very  little  measurement  of 
VoIP  traffic  has  been  carried  out,  a  prerequisite  to  characterization  and  control.  We  obtained  such  data  as 
part  of  our  study.  For  best-effort  traffic  the  headers  were  IP  and  TCP  (or  UDP).  For  voice  over  IP  traffic,  the 
headers  were  IP,  UDP,  and  RTP 

Task  2:  Statistical  Analysis,  Data  Mining,  and  Machine  Learning  for  Network  Characterization 
and  Modeling 

The  data  from  our  measurement  plans  were  studied  extensively  in  the  S-Net  environment.  We  determined 
variables  that  characterize  network  behavior  and  developed  models  for  network  traffic.  The  models  can 
provide  traffic  generation  for  network  simulators. 

Task  3:  Automated  Control 

We  developed  an  automated  control  method  for  network  self-management  based  on  the  above  variables  for 
network  characterization.  The  performance  of  the  control  methods  were  studied  through  simulation  methods 
and  analytical  mathematical  methods. 


5  Overview:  Key  Personnel 

1.  Supported  by  Grant. 

William  S.  Cleveland,  Professor  of  Statistics  and  Professor  of  Computer  Science,  Purdue  University 
Michael  Zhu,  Assistant  Professor  of  Statistics,  Purdue  University 
Hui  Chen,  Graduate  Student,  Statistics  Department,  Purdue  University 

2.  Substantial  work  was  carried  out  by  personnel  not  supported  by  the  grant. 

Bowei  Xi,  Assistant  Professor  of  Statistics,  Purdue  University 
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Douglas  Crabill,  System  Administrator,  Statistics  Department,  Purdue  University 
Scott  Ballew,  Lead  Network  Administrator,  Information  Technology,  Purdue  University 
Joerg  Micheel,  Head,  Passive  Measurement,  National  Laboratory  for  Applied  Network  Research 
Jin  Cao,  Member  of  Technical  Staff,  Bell  Labs 

Thomas  Telkamp,  Director  of  Research  and  Advanced  Development,  Global  Crossing 
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6  Summary  of  Technical  Results:  Data  Collection 


6.1  VoIP  Collection 

We  deployed  a  monitor  on  a  100-megabit  VoIP  Ethernet  link  on  the  Global  Crossing  Network.  The  link 
carries  VoIP  traffic  in  two  directions  between  a  Sonus  gateway  and  a  first  router  on  the  GBLX  IP  network. 
The  hardware  is  a  100-mbps  Ethernet  DAG  network  card  that  carries  out  measurement  on  the  card  and  can 
measure  at  full  line  speed  to  provide  highly  accurate  time  stamps.  The  DAG  card  timestamps  each  packet 
and  captures  the  IP/UDP/RTP  headers.  We  collected  data  for  48  hours. 

In  addition,  we  obtained  call  detail  records  for  each  call  during  the  first  24  hours  of  the  packet  trace.  The 
detail  records  let  us  see  unequivocally  the  caller  and  called  directions,  the  destinations  of  the  calls,  and 
categories  of  the  nature  of  the  call  terminations.  The  call  detail  records  provide  information  that  aid  greatly 
the  analysis  of  the  packet  traces. 


6.2  Best-Effort  Collection 

We  obtained  a  monitor  from  the  National  Laboratory  for  Applied  Network  Research  (NLANR)  for  packet 
trace  collection  on  the  1  gbps  link  that  carries  both  directions  of  all  traffic  between  the  Purdue  University 
campus  and  the  rest  of  the  Internet.  The  high-end  monitor  is  a  linux  box  with  a  monitor  operating  system 
together  with  a  light  PC  in  the  box  that  can  reboot  the  monitor  operating  system  and  allow  increased  remote 
maintenance.  The  box  also  contains  a  1-gbps  DAG  card  that  does  timestamping  and  packet  header  capture. 
Joerg  Micheel  of  NLANR  flew  to  the  West  Lafayette  campus  to  help  Scott  Ballew  and  Doug  Crabill  in  the 
installation.  By  agreeing  to  become  part  of  the  NLANR  measurement  program,  which  means  we  allow  data 
collection  in  paid  for  the  NLANR  database,  the  box  and  travel  for  Micheel  was  paid  for  by  the  NSL  grant 
that  supports  the  NLANR  program. 


6.3  Database  from  Previous  Collection  Efforts 

In  addition,  we  built  a  database  of  best  effort  packet  traces  from  6  Internet  links.  This  live  stream  database 
consists  of  349  streams,  90  sec  or  5  min  in  duration,  from  6  links  that  we  name  BELL,  NZIX,  AIX1,  AIX2, 
MLN1,  and  MLN2.  The  measured  streams  have  negligible  delay  on  the  link  input  router.  The  mean  number 
of  simultaneous  active  connections,  c,  ranges  from  49  connections  to  18,976  connections.  The  traffic  bit 
rate,  r,  ranges  from  1.00  megabits/sec  to  348  megabits/sec. 

BELL  is  a  100  megabit/sec  link  in  Murray  Hill,  NJ,  USA  connecting  a  Bell  Labs  local  network  of  about 
3000  hosts  to  the  rest  of  the  Internet.  The  transmission  is  half-duplex,  so  both  directions,  in  and  out,  are 
multiplexed  and  carried  on  the  same  link,  and  a  stream  is  the  multiplexing  of  both  directions;  but  to  keep 
the  variable  c  commensurate  for  all  of  the  six  links,  the  two  directions  for  each  connection  is  counted  as 
two.  In  this  presentation  we  use  195  BELL  traces,  each  5  min  in  length.  NZIX  is  the  100  megabit/sec 
New  Zealand  Internet  exchange  hosted  by  the  ITS  department  at  the  University  of  Waikato,  Hamilton,  New 
Zealand,  that  served  as  a  peering  point  among  a  number  of  major  New  Zealand  Internet  Service  Providers 
at  the  time  of  data  collection.  All  arriving  packets  from  the  input/output  ports  on  the  switch  are  mirrored, 
multiplexed,  and  sent  to  a  port  where  they  are  measured.  Because  all  connections  have  two  directions  at  the 
exchange,  each  connection  counts  2  as  for  BELL.  In  this  presentation  we  use  84  NZIX  traces,  each  5  min  in 
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length.  AIX1  and  AIX2  arc  two  separate  622  megabit/sec  OC12  packet-over-sonet  links,  each  carrying  one 
direction  of  the  traffic  between  NASA  Ames  and  the  MAE-West  Internet  exchange.  In  this  presentation  we 
use  23  AIX1  and  23  AIX2  traces,  each  90  sec  in  length.  AIX1  and  AIX2  streams  were  collected  as  paid  of 
a  project  at  the  National  Laboratory  for  Applied  Network  Research  where  the  data  arc  collected  in  blocks 
of  90  sec.  MFN1  and  MFN2  arc  two  separate  2.5  gigabit/sec  OC48  packet-over-sonet  links  on  the  network 
of  the  service  provider  Metropolitan  Fiber  Network;  each  link  carries  one  direction  of  traffic  between  San 
Jose,  California,  USA  and  Seattle,  Washington,  USA. 

We  also  downloaded  from  NLANR  data  collected  on  the  OC3  link  that  connects  the  University  of  Leipzig 
network  to  the  rest  of  the  Internet. 

7  Summary  of  Technical  Results:  Statistical  Analysis,  Data  Mining,  and 
Machine  Learning  for  Network  Characterization  and  Modeling 

7.1  Algorithms  for  Data  Processing  and  Strategies  of  Analysis 

We  purchased  and  installed  a  high-end  two-processor  linux  box  for  analysis,  learning,  mining,  and  modeling 
of  the  trace  data.  We  installed  the  S-Net  system  for  graphics  and  data  analysis  on  the  box. 

Algorithms  were  written  in  C,  Perl,  and  the  S  language  for  graphics  and  data  analysis  to  process  the  VoIP 
measurements.  The  algorithms  represent  a  a  dataflow  that  progresses  from  raw  packet  trace  files,  to  primary 
and  secondary  UNIX  flatfile  databases,  and  then  to  S  objects.  Our  analysis  is  canned  out  within  the  S 
environment  to  allow  comprehensive  study. 

An  important  part  of  the  strategy  for  study  is  time  blocking.  We  broke  the  trace  and  CDR  data  into  time 
blocks,  and  applied  the  same  analysis  to  all  blocks.  We  used  5  minute  blocks  to  guarantee  stationary  of 
traffic  processes  within  each  interval  and  to  create  object  sizes  that  were  not  to  large  for  efficient  computa¬ 
tion. 


7.2  Analysis  for  VoIP 

We  used  tools  of  statistical  analysis,  data  mining,  and  machine  learning  to  study  the  characteristics  of  the 
VoIP  packet  trace  data.  Most  important  in  this  exploratory,  or  first-study  phase,  were  a  host  of  visualization 
tools. 

We  discovered  that  there  were  a  very  large  number  of  packet  level  calls  with  no  CDR  records.  They  arc  all 
very  short.  We  studied  their  characteristics  to  see  if  they  had  anomalous  behavior.  They  spread  out  across 
time  and  space  in  proportion  to  the  calls  with  CDR  records.  We  concluded  that  certain  calls  short  enough 
not  to  generate  revenue  arc  not  saved  in  the  CDR  records. 

The  study  consisted  of  a  wide  range  of  topics  involving  the  VoIP  traffic  characteristics  and  the  quality 
performance  of  the  Global  Crossing  system 

•  call  arrival  process,  intensity  and  detailed  statistical  behavior 

•  call  characteristics  such  as  numbers  of  attempts  (without  connection)  and  connections,  call  durations, 
and  detailed  behavior  of  packet-level  characteristics  during  ringing  period  vs.  during  talking  period 
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•  packet  jitter 

•  aggregate  packet  traffic  statistical  properties  of  inter-arrival  times  and  packet  counts  in  20  ms  intervals 

•  silence  suppression 

A  number  of  results  came  from  our  analysis. 

•  call  inter-arrivals  have  to  a  very  good  approximation  an  exponential  marginal  distribution 

•  the  call  inter-arrivals  have  a  small  amount  of  short-term  positive  autocorrelation 

•  the  call  durations  have  a  Pareto  distribution,  which  breaks  with  the  past  exponential  behavior,  caused 
because  today’s  call  contain  faxes  and  computer  logins,  not  just  the  voice  conversations  of  the  past 

•  a  very  large  fraction  of  calls  arc  call  attempts 

•  patterns  in  the  ringing  make  us  believe  we  can  develop  a  classification  algorithm  for  determining  the 
ringing  period 

•  the  level  of  jitter  is  very  low,  about  ±150//sec. 

•  the  packet  arrival  process  is  long-range  dependent,  caused  by  the  Pareto  marginal  distribution  of  the 
call  durations;  these  statistical  properties  play  an  important  role  in  the  automated  control  rules  for 
bandwidth  allocation  that  we  arc  developing. 


7.3  FSD  Models  for  Best-Effort  Traffic 

The  statistical  properties  of  best-effort  traffic,  the  packet  arrival  process,  with  its  long-range  dependence, 
has  not  reached  a  consensus  with  different  camps  arguing  for  different  views. 

Almost  all  analysis  in  the  past  has  consisted  of  the  study  of  counts  of  packet  arrivals  in  a  fixed  interval  of 
time  such  as  1ms  or  10  ms.  This  is  not  sufficient  to  understand  the  arrival  process  because  moving  to  a 
count  is  an  information  reduction  method  to  simplify  analysis.  Instead,  we  studied  and  modeled  all  of  the 
information,  the  packet  arrival  times. 

We  developed  a  new  class  of  models,  fractional  sum-difference  (FSD)  models,  for  the  inter-arrival  times. 
The  FSD  models,  which  arc  non-Gaussian  and  long-range  dependent,  provide  excellent  fits  to  the  inter¬ 
arrivals.  We  carried  out  extensive  validation  to  demonstrate  this.  Our  models  provide  a  bridge  between 
the  different  camps,  explaining  and  integrating  results  on  multifractal  modeling,  long-range  dependence, 
fractional  Brownian  motion,  and  convergence  to  Poisson  processes. 

In  addition,  our  models  can  be  used  for  open-loop  generation  of  background  traffic  in  network  simulators. 
This  represents  the  first  such  generation  that  properly  recreates  the  properties  of  real  traffic. 

Suppose  xv  for  v  =  1,  2, ...  is  a  stationary  time  series  with  marginal  cumulative  distribution  function 
F(x;  0)  where  (f)  is  a  vector  of  unknown  parameters.  Let  x*  =  H  (xv:  0)  be  a  transformation  of  xv  such  that 
the  marginal  distribution  of  x*  is  normal  with  mean  0  and  variance  1.  We  have  H (xv;  0)  =  G~  l(l7(.x;  0)), 
where  G(z)  is  the  cumulative  distribution  function  of  a  normal  random  variable  with  mean  0  and  variance 
1.  Next  we  suppose  x*  is  a  Gaussian  time  series  and  call  x*  the  “Gaussian  image”  of  xv. 

Suppose  x*  has  the  following  form: 

x*  =  v7!  -  0  sv  -F  Vo  nv, 
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where  sv  and  nv  arc  independent  of  one  another  and  each  has  mean  0  and  variance  1.  nv  is  Gaussian  white 
noise,  that  is,  an  independent  time  series.  sv  is  a  Gaussian  fractional  ARIMA  (Hosking  1981) 

(I  —  B)dsv  =  ev  +  eu-\ 

where  Bsv  =  s„_i,  0  <  d  <  0.5,  and  ev  is  Gaussian  white  noise  with  mean  0  and  variance 

2_  (i-d)r2(i-cQ 
ae  2r(i  -  2d)  ' 


xv  is  &  fractional  sum- difference  ( FSD )  time  series.  Its  Gaussian  image,  x*,  has  two  components,  fl  —  9sv 
is  the  long-range-dependent,  or  lrd,  component;  its  variance  is  1  —  0.  \/6  is  the  white-noise  component;  its 
variance  is  6. 


Let  px*  (/)  be  the  power  spectrum  of  the  x*.  Then 


Px * (/) 


(1  -  9)a2 


4  cos2  (7 r/) 
(4  sin2  (7 rf))d 


+  9 


for  0  <  /  <  0.5.  As  f  —y  0.5,  px*  (/)  decreases  monotonically  to  9.  As  /  — ►  0,  px*  (/)  goes  to  infinity  like 
sin~2d(7r/)  ~  f  2d\  one  outcome  of  long-range  dependence.  For  nonnegative  integer  lags  k,  let  rx*{k), 
rs{k ),  and  rn(k)  be  the  autocovariance  functions  of  x*,  sv,  and  nv,  respectively.  Because  the  three  series 
have  variance  1,  the  autocovariance  functions  are  also  the  autocorrelation  functions.  rs(k)  is  positive  and 
falls  off  like  k2d  1  as  k  increases,  another  outcome  of  long-range  dependence.  For  k  >  0,  rn{k)  =  0  and 


rx*(k)  =  (1  -  9)rs(k). 


As  6  — >  1 ,  x*  goes  to  white  noise:  px*  (/)  — >  1  and  rx*  (. k )  — >  0  for  k  >  0.  The  change  in  the  autocovariance 
function  and  power  spectrum  arc  instructive.  As  9  gets  closer  to  1,  the  rise  of  px*  (/)  near  /  =  0  is  always 
to  order  /~2a!,  and  the  rate  of  decay  of  rx*(k)  for  large  k  is  always  If 2,1  1 ;  but  the  ascent  of  px*(f)  at  the 
origin  begins  closer  and  closer  to  /  =  0  and  the  rx*{k)  get  uniformly  smaller  by  the  multiplicative  factor 
1  -  9. 

We  model  the  marginal  distribution  of  tv  by  a  Weibull  with  shape  A  and  scale  a,  a  family  with  two  unknown 
parameters.  Estimates  of  A  arc  almost  always  less  than  1.  The  Weibull  provides  an  excellent  approximation 
of  the  sample  marginal  distribution  of  the  tv  except  that  the  smallest  3%  to  5%  of  the  sample  distribution  is 
truncated  to  a  nearly  constant  value  due  to  certain  network  transmission  properties. 

The  marginal  distribution  of  qv  is  modeled  as  follows.  While  packets  less  than  40  bytes  can  occur,  it  is 
sufficiently  rare  that  we  ignore  this  and  suppose  40  <  qv  <  1500.  First,  we  provide  for  A  atoms  at  sizes 
(j>\  . . .  0A  such  as  40  bytes,  512  bytes,  576  bytes,  and  1500  bytes  that  arc  commonly  occurring  sizes;  the 
atom  probabilities  arc  ]  For  the  remaining  sizes,  we  divided  the  interval  [40,  1500]  bytes  into 

C  sub-intervals  using  C  —  1  distinct  break  points  with  values  that  are  greater  than  40  bytes  and  less  than 
1500  bytes,  . . .  4^c-i-  For  each  of  the  C  sub-intervals,  the  size  distribution  has  an  equal  probability 
for  the  remaining  sizes  (excluding  the  atoms)  in  the  sub-interval;  the  total  probabilities  for  the  sub-intervals 
are  cf>\  , . . . ,  f  c  .  Typically,  with  just  3  atoms  at  40  bytes,  576  bytes,  and  1500  bytes,  and  with  just  2  break 
points  at  50  bytes  and  200  bytes,  we  get  an  excellent  approximation  of  the  marginal  distribution. 

The  transformed  time  series  t*  and  q*  appeal-  to  be  quite  close  to  Gaussian  processes.  Some  small  amount 
of  non-Gaussian  behavior  is  still  present  but  it  is  minor.  The  autocorrelation  structure  of  these  Gaussian 
images  are  very  well  fitted  by  the  FSD  autocorrelation  structure. 


The  parameters  of  the  FSD  model  are  the  following: 

•  qv  marginal  distribution:  A  atom  probabilities  (pf1  at  A  sizes  C  —  1  break  points  <j)f^  and  C 
sub-interval  probabilities  <j>\ 

•  tv  marginal  distribution:  shape  A  and  scale  a 

•  q*  time  dependence:  fractional  difference  coefficient  d/'11  and  white-noise  variance  0(q) 

•  t*  time  dependence:  fractional  difference  coefficient  diL)  and  white-noise  variance  0(*). 

We  found  that  the  d-A  and  d ^  do  not  depend  on  c;  this  is  based  on  empirical  study,  and  it  is  supported 
by  theory.  The  estimated  values  arc  0.410  and  0.411,  respectively.  We  take  the  value  of  each  of  these  two 
parameters  to  be  0.41.  We  found  that  as  c  increases,  estimates  of  A,  0iqK  and  all  tend  toward  1.  This 
means  the  tv  tend  to  a  independent  exponentials,  a  Poisson  process  and  the  qv  tend  tow  ard  independence.  In 
other  words,  the  statistical  models  account  for  the  change  in  the  tv  and  qv  and  c  increases  that  was  discussed 
earlier.  We  estimated  these  three  parameters  and  a  by  partial  likelihood  methods  with  d/'11  and  fixed  to 
0.41.  The  marginal  distribution  of  qv  on  a  given  link  does  not  change  with  c,  but  it  does  change  from  link 
to  link.  To  generate  traffic,  we  must  specify  the  atom  and  sub-interval  probabilities.  This  provides  a  mean 
packet  size  fi(qy  measured  in  bits/packet. 


8  Automated  Control:  Bandwidth  Estimation  for  Best-Effort  Traffi  c 

8.1  Problem  Formulation 

Suppose  the  packet  stream  —  packet  arrival  times  and  sizes  —  arriving  for  transmission  on  an  Internet  link 
is  best-effort  traffic  with  bit  rate  r  bits/sec  and  number  of  simultaneous  active  connections,  c.  Suppose  the 
link  input  buffer  is  large  enough  that  packet  loss  is  negligible.  Our  goal  is  to  estimate  the  quality-of-service 
(QoS)  bandwidth  (3  in  bits/sec,  or  equivalently,  the  QoS  utilization  u  =  t/  ft,  that  satisfies  QoS  criteria  for 
the  packet  queueing  delay  in  the  link  input  buffer.  The  criteria  arc  a  delay,  5  sec,  and  the  probability,  u,  that 
the  delay  for  a  packet  exceeds  5. 


8.2  Other  Work  on  the  Problem 

There  is  a  wide  literature  on  the  bandwidth  estimation  problem.  Much  of  it  is  theoretical  —  mathematical 
results  that  derive  properties  of  queueing  systems.  A  smaller  literature  is  empirical  in  nature,  based  on 
simulations  with  packet  stream  inputs  from  measurements  on  live  links  or  from  models  for  traffic.  The 
classical  Erlang  Delay  Formula  provides  a  simple  formula  that  can  be  used  for  estimation  of  traffic  streams 
that  in  theory  have  Poisson  arrivals  and  i.i.d.  exponential  sizes.  Best-effort  traffic  is  much  more  complex,  it 
is  nonlinear  and  long-range  dependent,  and  there  has  been  to  date  no  simple,  validated  formula  for  it. 


8.3  Principal  Result:  The  Best-Effort  Delay  Model 

The  principal  result  of  this  report  is  a  statistical  model  that  provides  a  simple,  validated  formula  for  the 
estimation  of  bandwidth  for  best-effort  traffic  in  the  same  way  that  the  Erlang  Delay  Formula  does  for  the 
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Poisson-exponential  case.  The  model  has  been  validated  through  extensive  empirical  study  and  through 
consistency  with  certain  theoretical  properties  of  queueing. 

The  model  consists  of  the  Best-Effort  Delay  Formula  plus  random  variation: 

logit 2{u)  =o  +  oc  log 2(c)  +  otS  log 2{tS)  +  ow(-  log2 (  log2(w)))  +  ip. 

ip  is  a  random  error  variable  with  mean  0  and  median  absolute  deviation  mg  (ip),  which  depends  on  <5;  log2 
is  the  log  base  2;  and  logit2(it)  =  log2(tt/(l  —  u)).  The  distribution  of  0.691  ip /ms (ip)  is  a  /-distribution 
with  15  degrees  of  freedom.  Estimates  of  the  coefficients  of  the  model  arc 

d  =  —8.933  dc  =  0.420  oT$  =  0.444  dw  =  0.893. 

mg(ip)  is  modeled  as  a  function  of  6:  log 2(m$(ip))  is  a  linear  function  of  logy  (4)  plus  random  variation. 
The  estimate  of  the  intercept  of  the  line  is  -0.481,  the  estimate  of  the  linear  coefficient  of  the  line  is  0.166, 
and  the  estimate  of  the  standard  error  is  0.189.  The  bit  rate  r  is  equal  to  cy;,  where  75  is  the  connection  bit 
rate  in  bits/sec/connection.  So  the  Best-Effort  Delay  Formula  can  also  be  written  as 

logit2(«)  =  o  +  (oc  +  orS)  log2(c)  +  oTs  log2(y bS)  +  ow(-  log2 (  log2(w))). 

In  this  form  we  see  the  action  of  the  amount  of  multiplexing  of  connections  as  measured  by  c,  and  the  end- 
to-end  connection  speed  as  measured  by  75.  An  increase  in  either  results  in  an  increase  in  the  utilization  of 
a  link. 


The  Best-Effort  Delay  Model  is  used  to  estimate  the  bandwidth  required  to  carry  best-effort  traffic  given  S, 
u,  t,  and  c.  The  QoS  logit  utilization  is  estimated  by 

i  =  —8.933  +  0.420  log2(c)  +  .444  log2(r<5)  +  0.893(—  log2(—  log2(u;))), 


so  the  QoS  utilization  is  estimated  by 


2l 

l  +  2r 


The  corresponding  estimated  bandwidth  is  t/u.  For  such  an  estimate  there  is  a  50%  chance  of  being  too 
large  and  a  50%  chance  of  being  too  small.  We  might  however,  use  a  more  conservative  estimate  that 
provides  a  much  smaller  chance  of  too  little  bandwidth.  Let 


msW  =  2-°-481+0-1661°g2(5) 


be  the  estimate  of  m(8).  Let  tirjp)  be  the  lower  100p%  percentage  point  of  a  /-distribution  with  15  degrees 
of  freedom,  where  p  is  small,  say  0.05.  Let 

£(p)  =  £-m(<y)ii5(p)/0.691 


Then 


u(p)  = 


2  Z(p) 


1  +  2  hp)  ’ 

is  a  conservative  utilization  estimate,  the  lower  limit  of  a  100p%  tolerance  interval  for  the  QoS  utilization. 
The  corresponding  estimated  bandwidth  is  t /u(p). 
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8.4  Methods 


The  Best-Effort  Delay  Model  was  built,  in  part,  from  queueing  theory.  Certain  predictor  variables  were 
suggested  by  the  Erlang  Delay  Formula.  Theory  prescribes  certain  behavior  as  r,  c,  or  75  increase,  resulting 
in  rate  gains,  multiplexing  gains,  or  fast-forward  invariance,  and  the  model  was  constructed  to  reproduce  the 
behavior. 

The  Best-Effort  Delay  Model  was  built,  in  part,  from  results  of  queueing  simulations  with  traffic  stream 
inputs  of  two  types,  live  and  synthetic.  The  live  streams  arc  measurements  of  packet  arrivals  and  sizes  for 
349  intervals,  90  sec  or  5  min  in  duration,  from  6  Internet  links.  The  synthetic  streams  arc  arrivals  and 
sizes  generated  by  recently  developed  FSD  time  series  models  for  the  arrivals  and  sizes  of  best-effort  traffic. 
Each  of  the  live  streams  was  fitted  by  two  FSD  models,  one  for  the  inter-arrivals  and  one  for  the  sizes,  and 
a  synthetic  stream  of  5  min  was  generated  by  the  models.  The  generated  inter-arrivals  are  independent  of 
the  generated  sizes,  which  is  what  we  found  in  the  live  data.  The  result  is  349  synthetic  streams  that  match 
the  statistical  properties  collectively  of  the  live  streams.  For  each  live  or  synthetic  stream,  we  carried  out 
25  runs,  each  with  a  number  of  simulations.  For  each  run  we  picked  a  delay  6  and  a  delay  probability  uj\ 
simulations  were  carried  out  to  find  the  QoS  bandwidth  (3,  the  bandwidth  that  results  in  delay  probability 
c 0  for  5.  This  also  yields  a  QoS  utilization  u  =  t/(3.  We  used  5  delays  —  0.001  sec,  0.005  sec,  0.010 
sec,  0.050  sec,  0.100  sec  —  and  5  delay  probabilities  —  0.001,  0.005,  0.01,  0.02,  and  0.05  —  employing 
all  25  combinations  of  the  two  delay  criteria.  The  queueing  simulation  results  in  delay  data:  values  of  five 
variables  —  QoS  utilization  u,  delay  6,  delay  probability  o>,  the  mean  number  of  active  connections  of  the 
traffic  c,  and  the  traffic  bit  rate  r.  The  delay  data  were  used  in  the  model  building. 


8.5  Validity  and  Applicability 

Extensive  data  exploration  with  visualization  tools,  some  shown  here,  demonstrate  that  the  Best-Effort  Delay 
Model  fits  the  simulation  delay  data.  This,  of  course,  is  necessary  for  the  model  to  be  valid.  In  addition, 
validity  is  supported  by  the  model  reproducing  the  theoretical  queueing  properties  as  just  discussed. 

The  validity  of  the  Best-Effort  Delay  Model  depends  on  the  validity  of  the  traffic  streams  used  as  inputs 
to  the  queueing  simulation;  that  is,  the  packet  streams  must  reproduce  the  statistical  properties  of  best- 
effort  streams.  Of  course,  the  live  streams  of  the  study  do  so  because  they  arc  best-effort  traffic.  Extensive 
validation  has  shown  that  the  FSD  models,  used  to  generate  the  packet  streams  here,  provide  excellent  fits 
to  best-effort  packet  streams  when  c  is  above  about  64  connections,  which  for  a  link  where  7*,  is  about  214 
bits/sec/connection  means  r  is  above  about  1  megabit/sec.  For  this  reason,  only  traffic  streams  with  r  greater 
than  this  rate  are  used  in  the  study,  and  the  Best-Effort  Delay  Model  is  valid  above  this  rate. 

The  results  arc  only  valid  for  links  with  a  large  enough  buffer  that  the  packet  loss  is  negligible.  We  have 
used  open-loop  study,  which  does  not  provide  for  the  TCP  feedback  that  occurs  when  loss  is  significant. 

There  is  also  a  practical  restriction  on  applicability.  We  have  taken  the  range  of  our  study  to  include  traffic 
bit  rates  as  low  as  about  1  megabit/sec.  We  have  done  this  simply  because  we  can  do  so  and  achieve  valid 
results.  But  even  for  the  least  stringent  of  our  delay  criteria,  6  =  0. 1  sec  delay  and  u  =  0.05  delay  probability, 
the  utilizations  arc  low  for  rates  in  the  range  of  1  megabit/sec  to  5  megabits/sec.  This  might  well  be  judged 
to  be  too  small  a  utilization  to  be  practical;  if  so,  it  might  mean  that  the  what  must  be  sacrificed  is  the 
negligible  packet  loss,  which  means  that  a  QoS  study  at  the  very  low  traffic  bit  rates  needs  to  take  account 
of  TCP  feedback. 
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One  outcome  of  the  dependence  of  the  bandwidth  estimation  on  the  traffic  statistics  is  that  our  solution  for 
best-effort  traffic  would  not  apply  to  other  forms  of  Internet  traffic  that  do  not  share  the  best-effort  statistical 
properties.  One  example  is  voice  traffic. 

Finally,  the  Best-Effort  Delay  Model  provides  an  estimation  of  bandwidth  in  isolation  without  considering 
other  network  factors.  A  major  factor  in  network  design  is  link  failures.  Redundancy  needs  to  be  built  into 
the  system.  So  an  estimate  of  bandwidth  from  the  model  for  a  link  based  on  the  normal  link  traffic  may  be 
reduced  to  provide  the  redundancy.  But  the  model  still  plays  a  role  because  the  bandwidth  must  be  chosen 
based  on  link  traffic,  but  now  the  traffic  in  the  event  of  a  failure  elsewhere. 


9  Automated  Control:  Bandwidth  Estimation  for  VoIP 


Today,  no  automated  methods  exist  for  determining  how  many  calls  can  be  put  on  a  VoIP  link  and  maintain 
QoS  criteria  for  delay  and  packet  jitter.  We  designed  a  queueing  simulation  and  modeling  mechanism  to 
determine  a  statistical  process-control  approach  that  would  allow  automation  provisioning  of  traffic  on  a 
link. 

In  the  simulation,  calls  arrive  as  a  Poisson  process,  a  reasonable  approximation  to  the  observed  behavior  of 
the  arrivals  process.  For  each  arrival,  we  sample  with  replacement  from  our  call  database,  about  300,000 
calls,  and  use  the  observed  packet  process  of  each  sampled  call  in  the  simulation. 

In  the  simulation  we  fixed  the  traffic  call  rate  and  vary  the  link  bandwidth  (link  speed)  so  that  the  utilization 
ranges  from  30%  to  99%,  and  study  the  resulting  queueing  delay  and  jitter  of  the  packets.  We  built  statistical 
models  for  the  delay  and  jitter  based  on  the  utilization,  the  number  of  simultaneous  active  calls  and  the 
bitrate.  The  models  provide  the  requisite  statistical  process  control. 

We  studied  our  simulation  runs  and  discovered  that  there  arc  instabilities  when  the  utilization  gets  beyond  a 
certain  limit,  which  makes  delay  estimation  problematic.  The  limit  decreases  and  the  traffic  rate  decreases. 
The  phenomenon  means  the  network  cannot  operate  at  these  utilizations,  so  they  are  not  relevant.  We  cannot 
leave  them  in  the  data  to  be  modeled  because  they  depart  from  standard  conditions,  so  we  will  eliminated 
them  from  the  study,  using  machine  learning  rules  for  the  elimination. 


10  Work  Under  The  Current  Contract 


This  report  describes  the  work  done  under  the  above  federal  contract  as  well  as  previous  work  done  under 
federal  contract  No.  F30602-02-2-C0093  awarded  to  Bell  Fabs  because  the  work  under  the  new  contract 
was  a  continuation  of  the  work  under  the  previous  contract. 

The  following  describes  that  portion  of  the  work  completed  under  the  new  contract: 

Task  1 :  Packet-Fevel  Measurement 

•  Newark  VoIP  packet  trace  data  collection  and  data  processing 

•  Feipzig  best-effort  traffic  processing 

•  installation  of  NFANR  packet  trace  collector  on  Purdue  Internet  access  link  for  best-effort  traffic 
measurement 
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Task  2:  Statistical  Analysis,  Data  Mining,  and  Machine  Learning  for  Network  Characterization  and  Model¬ 
ing 


•  all  analysis  of  the  Newark  VoIP  data 

•  all  work  on  the  integration  of  best-effort  traffic  models  through  the  understanding  provided  by  the 
fractional  sum-difference  models 


Task  3:  Automated  Control 

•  all  simulations  for  bandwidth  estimation  for  VoIP  traffic  and  ensuing  analysis  and  modeling  of  results 

•  simulations  for  best-effort  traffic  were  completely  redesigned  and  carried  out,  and  new  results  ana¬ 
lyzed  and  modeled 


11  Detailed  Technical  Report 

11.1  Introduction:  Contents  of  the  Paper 

The  Internet  is  a  world-wide  computer  network;  at  any  given  moment,  a  vast  number  of  pairs  of  hosts  are 
transferring  files  one  to  the  other.  Each  transferred  file  is  broken  up  into  packets  that  arc  sent  along  a  path 
across  the  Internet  consisting  of  links  and  nodes.  The  first  node  is  the  sending  host;  a  packet  exits  the  host 
and  travels  along  a  link  (fiber,  wire,  cable,  or  air)  to  a  first  router  node,  then  over  a  link  to  a  second  router 
node,  and  so  forth  until  a  last  router  sends  the  packet  to  a  receiving  host  node  over  a  final  link. 

The  packet  traffic  arriving  for  transmission  on  an  Internet  link  is  a  stream:  a  sequence  of  packets  with  arrival 
times  (sec)  and  sizes  (bytes  or  bits).  The  packets  come  from  pairs  of  hosts  using  the  link  for  their  transfers; 
that  is,  the  link  lies  on  the  path  from  one  host  to  another  for  each  of  a  collection  of  pairs  of  hosts.  When  a 
packet  arrives  for  transmission  on  a  link,  it  enters  a  buffer  (bits)  where  it  must  wait  if  there  are  other  packets 
waiting  for  transmission  or  if  a  packet  is  in  service,  that  is,  in  the  process  of  moving  out  of  the  buffer  onto 
the  link.  If  the  buffer  is  full,  the  packet  is  dropped. 

A  link  has  a  bandwidth  (bits/sec),  the  rate  at  which  the  bits  of  a  packet  arc  put  on  the  link.  Over  an  interval 
of  time  during  which  the  traffic  is  stationary,  the  packets  arrive  for  transmission  at  a  certain  rate,  the  traffic 
bit  rate  (bits/sec),  which  is  defined,  formally,  to  be  the  mean  of  the  packets  sizes  (bits)  divided  by  the  mean 
packet  inter-arrival  time  (sec),  but  this  is  approximately  the  mean  number  of  arriving  bits  over  the  interval 
divided  by  the  interval  length  (sec).  Over  the  interval  there  is  a  mean  simultaneous  active  connection  load, 
the  mean  number  of  source-destination  pairs  of  hosts  actively  sending  packets  over  the  link.  The  utilization 
of  the  link  is  the  traffic  bit  rate  divided  by  the  bandwidth;  it  measures  the  traffic  rate  relative  to  the  capacity 
of  the  link. 

This  article  presents  results  on  a  fundamental  problem  of  engineering  the  Internet.  What  link  bandwidth  is 
needed  to  accommodate  traffic  with  a  certain  bit  rate  and  ensure  that  the  transmission  on  the  link  maintains 
quality-of-service,  or  QoS,  criteria?  Finding  the  QoS  bandwidth  must  be  faced  in  setting  up  every  link  on 
the  Internet,  from  the  low-bandwidth  links  connecting  the  computers  of  a  home  user  to  the  high-bandwidth 
links  of  a  major  Internet  service  provider.  Our  approach  to  solving  the  bandwidth  estimation  problem  is  to 
use  queueing  theory  and  queueing  simulations  to  build  a  model  for  the  QoS  bandwidth.  The  traffic  inputs 
are  live  streams  from  measurements  of  live  links  and  synthetic  streams  from  statistical  models  for  traffic 
streams. 
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Section  11.2  describes  Internet  TCP/IP  transmission  technology,  which  governs  almost  all  computer  net¬ 
working  today  —  for  example,  the  networks  of  Internet  service  providers,  universities,  companies,  and 
homes.  Section  1 1.2  also  describes  the  buffer  queueing  process  and  its  affect  on  the  QoS  of  file  transfer. 

Section  11.3  formulates  the  particular  version  of  the  bandwidth  estimation  problem  that  is  addressed  here, 
discusses  why  the  statistical  properties  of  the  packet  streams  are  so  critical  to  bandwidth  estimation,  and 
oudines  how  we  use  queueing  simulations  to  study  the  problem.  We  study  “best-effort”  Internet  traffic 
streams  because  they  arc  the  predominant  type  of  traffic  on  Internet  links  today.  The  QoS  criteria  for 
best-effort  streams  are  the  packet  loss  (fraction  of  lost  packets),  the  queueing  delay  (sec),  and  the  delay 
probability  (probability  of  a  packet  exceeding  the  delay).  We  suppose  that  the  link  packet  loss  is  negligible, 
and  find  the  QoS  bandwidth  required  for  a  packet  stream  of  a  certain  load  that  satisfies  the  delay  and  the 
delay  probability. 

Section  11.4  describes  fractional  sum-difference  (FSD)  time  series  models,  which  arc  used  to  generate  the 
synthetic  streams  for  the  queueing  simulations.  The  FSD  models,  a  new  class  of  non-Gaussian,  long-range 
dependent  time  series  models,  provide  excellent  fits  to  packet  size  time  series  and  to  packet  inter-arrival 
time  series.  The  validation  of  the  FSD  models  is  critical  to  this  study.  The  validity  of  our  solution  to  the 
bandwidth  estimation  problem  depends  on  having  traffic  inputs  to  the  queueing  that  reproduce  the  statistical 
properties  of  best-effort  traffic.  Of  course,  the  live  data  have  the  properties,  but  we  need  assurance  that  the 
synthetic  data  do  as  well. 

Section  11.5  describes  the  live  packet  arrivals  and  sizes  and  the  synthetic  packet  arrivals  and  sizes  that  are 
generated  by  the  FSD  models.  Section  1 1.6  gives  the  details  of  the  simulations  and  the  resulting  delay  data: 
values  of  the  QoS  bandwidth,  delay,  delay  probability,  mean  number  of  active  host-pair  connections  of  the 
traffic,  and  traffic  bit  rate. 

Model  building,  based  on  the  simulation  delay  data  and  on  queueing  theory,  begins  in  Section  11.7.  To 
do  the  model  building  and  diagnostics,  we  exploit  the  structure  of  the  delay  data  —  utilizations  for  all 
combinations  of  delay  and  delay  probability  for  each  stream,  live  or  synthetic.  We  develop  an  initial  model 
that  relates,  for  each  stream,  the  QoS  utilization  (bit  rate  divided  by  the  QoS  bandwidth)  to  the  delay  and 
delay  probability.  We  find  a  transformation  for  the  utilization  for  which  the  functional  dependence  on  the 
delay  and  delay  probability  does  not  change  with  the  stream.  There  is  also  an  additive  stream  coefficient 
that  varies  across  streams,  characterizing  the  statistical  properties  of  each  stream.  This  stream-coefficient 
delay  model  cannot  be  used  for  bandwidth  estimation  because  the  stream  coefficient  would  not  be  known  in 
practice. 

Next  we  add  two  variables  to  the  model  that  measure  the  statistical  properties  of  the  streams  and  that  can 
be  specified  or  measured  in  practice  —  the  traffic  bit  rate  and  the  number  of  simultaneous  active  host-pair 
connections  on  the  link  —  and  drop  the  stream  coefficients.  In  effect  we  have  modeled  the  coefficients. 
The  result  is  the  Best-Effort  Delay  Model:  a  Best-Effort  Delay  Formula  for  the  utilization  as  a  function  of 
(1)  the  delay,  (2)  the  delay  probability,  (3)  the  traffic  bit  rate,  and  (4)  the  mean  number  of  active  host-pair 
connections  of  the  traffic,  plus  a  random  error  term. 

Section  11.8  presents  a  method  for  bandwidth  estimation  that  starts  with  the  value  from  the  Best-Effort 
Delay  Formula,  and  then  uses  the  error  distribution  of  the  Best-Effort  Delay  Model  to  find  a  tolerance 
interval  whose  minimum  value  provides  an  conservative  estimate  with  a  low  probability  of  being  too  small. 

Section  11.9  discusses  previous  work  on  bandwidth  estimation,  and  how  it  differs  from  the  work  here. 
Section  11.10  is  an  extended  abstract.  Readers  who  seek  just  results  can  proceed  to  this  section;  those  not 
familial-  with  Internet  engineering  technology  might  want  to  first  read  Sections  1 1.2  and  1 1.3. 
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The  following  notation  is  used  throughout  the  article: 
Packet  Stream 


•  v  :  arrival  numbers  (number)  v  =  1  is  the  first  packet,  v  =  2  is  the  second  packet,  etc. 

•  av  :  arrival  times  (sec) 

•  tv  :  inter-arrival  times  (sec)  tv  =  av+  \  —  av 

•  qv  :  sizes  (bytes  or  bits). 

Traffic  Load 

•  c  :  mean  number  of  simultaneous  active  connections  (number) 

•  r  :  traffic  bit  rate  (bits/sec) 

•  7 p  :  connection  packet  rate  (packets/sec/connection) 

•  7b  :  connection  bit  rate  (bits/sec/connection). 

Bandwidth 

•  ft  :  bandwidth  (bits/sec) 

•  u  :  utilization  (fraction)  r  j ft. 

Queueing 

•  6  :  packet  delay  (sec) 

•  t o  :  delay  probability  (fraction). 

11.2  Internet  Technology 

The  Internet  is  a  computer  network  over  which  a  pair  of  host  computers  can  transfer  one  or  more  files  (Stevens 
1994).  Consider  the  downloading  of  a  Web  page,  which  is  often  made  up  of  more  than  one  file.  One  host,  the 
client,  sends  a  request  file  to  start  the  downloading  of  the  page.  Another  host,  the  server,  receives  the  request 
file  and  sends  back  a  first  response  file;  this  process  continues  until  all  of  the  response  files  necessary  to 
display  the  page  are  sent.  The  client  passes  the  received  response  files  to  a  browser  such  as  Netscape,  which 
then  displays  the  page  on  the  screen.  This  section  gives  information  about  some  of  the  Internet  engineering 
protocols  involved  in  such  file  transfer. 


11.2.1  Packet  Communications 

When  a  file  is  sent,  it  is  broken  up  into  packets  whose  sizes  are  1460  bytes  or  less.  The  packets  arc  sent 
from  the  source  host  to  the  destination  host  where  they  arc  reassembled  to  form  the  original  file.  They  travel 
along  a  path  across  the  Internet  that  consists  of  transmission  links  and  routers.  The  source  computer  is 
connected  to  a  first  router  by  a  transmission  link;  the  first  router  is  connected  to  a  second  router  by  another 
transmission  link,  and  so  forth.  A  router  has  input  links  and  output  links.  When  it  receives  a  packet  from  one 
of  its  input  links,  it  reads  the  destination  address  on  the  packet,  determines  which  of  the  routers  connected 
to  it  by  output  links  should  get  the  packet,  and  sends  out  the  packet  over  the  output  link  connected  to  that 
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router.  The  flight  across  the  Internet  ends  when  a  final  router  receives  the  packet  on  one  of  its  input  links 
and  sends  the  packet  to  the  destination  computer  over  one  of  its  output  links. 

The  two  hosts  establish  a  connection  to  carry  out  one  or  more  file  transfers.  The  connection  consists  of 
software  running  on  the  two  computers  that  manages  the  sending  and  receiving  of  packets.  The  software 
executes  an  Internet  transport  protocol,  a  detailed  prescription  for  how  the  sending  and  receiving  should 
work.  The  two  major  transport  protocols  arc  UDP  and  TCP.  UDP  just  sends  the  packets  out.  With  TCP, 
the  two  hosts  exchange  control  packets  that  manage  the  connection.  TCP  opens  the  connection,  closes  it, 
retransmits  packets  not  received  by  the  destination,  and  controls  the  rate  at  which  packets  arc  sent  based  on 
the  amount  of  retransmission  that  occurs.  The  transport  software  adds  a  header  to  each  packet  that  contains 
information  about  the  file  transfer.  The  header  is  20  bytes  for  TCP  and  8  bytes  for  UDP. 

Software  running  on  the  two  hosts  implements  another  network  protocol,  IP  (Internet  Protocol),  that  man¬ 
ages  the  involvement  of  the  two  hosts  in  the  routing  of  a  packet  across  the  Internet.  The  software  adds  a 
20-byte  IP  header  to  the  packet  with  information  needed  in  the  routing  such  as  the  source  host  IP  address 
and  the  destination  host  IP  address.  IP  epitomizes  the  conceptual  framework  that  underlies  Internet  packet 
transmission  technology.  The  networks  that  make  up  the  Internet  —  for  example,  the  networks  of  Internet 
service  providers,  universities,  companies,  and  homes  —  arc  often  referred  to  as  IP  networks,  although  today 
it  is  unnecessary  because  almost  all  computer  networking  is  IP,  a  public-domain  technology  that  defeated 
all  other  contenders,  including  the  proprietary  systems  of  big  computer  and  communications  companies. 


11.2.2  Link  Bandwidth 

The  links  along  the  path  between  the  source  and  destination  hosts  each  have  a  bandwidth  /3  in  bits/sec.  The 
bandwidth  refers  to  the  speed  at  which  the  bits  of  a  packet  arc  put  on  the  link  by  a  computer  or  router.  For  the 
link  connecting  a  home  computer  to  a  first  router,  (3  might  be  56  kilobits/sec  if  the  computer  uses  an  internal 
modem,  or  1.5  megabits/sec  if  there  is  a  broadband  connection,  a  cable  or  DSL  link.  The  link  connecting  a 
university  computer  to  a  first  router  might  be  10  megabits/sec,  100  megabits/sec,  or  1  gigabit/sec.  The  links 
on  the  core  network  of  a  major  Internet  service  provider  have  a  wide  range  of  bandwidths;  typical  values 
range  from  45  megabits/sec  to  10  gigabits/sec.  For  a  40  byte  packet,  which  is  320  bits,  it  takes  5.714  ms 
to  put  the  packet  on  a  56  kilobit/sec  link,  and  0.032  /is  to  put  it  on  a  10  gigabits/sec  link,  which  is  about 
1 80,000  times  faster.  Once  a  bit  is  put  on  the  link,  it  travels  down  the  link  at  the  speed  of  light. 


11.2.3  Active  Connections,  Statistical  Multiplexing,  and  Measures  of  Traffic  Loads 

At  any  given  moment,  an  Internet  link  has  a  number  of  simultaneous  active  connections;  this  is  the  number 
of  pairs  of  computers  connected  with  one  another  that  are  sending  packets  over  the  link.  The  packets  of 
the  different  connections  arc  intermingled  on  the  link;  for  example,  if  there  arc  three  active  connections,  the 
arrival  order  of  10  consecutive  packets  by  connection  number  might  be  1,  1,  2,  3,  1,  1,  3,  3,  2,  and  3.  The 
intermingling  is  referred  to  as  “statistical  multiplexing”.  On  a  link  that  connects  a  local  network  with  about 
500  users  there  might  be  300  active  connections  during  a  peak  period.  On  the  core  link  of  an  Internet  service 
provider  there  might  be  60,000  active  connections. 

During  an  interval  of  time  when  the  traffic  is  stationary,  there  is  a  mean  number  of  active  connections  c,  and 
a  traffic  bit  rate  r  in  bits/sec.  Let  /ip)  in  sec  be  the  mean  packet  inter-arrival  time,  and  let  H(q),  in  bits,  be 
the  mean  packet  size.  Then  the  packet  arrival  rate  per  connection  is  yp  =  c~’  packets/sec/connection. 
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The  bit  rate  per  connection  is  75  =  M(g)c_  Vp)1  =  TC~ '  bits/sec/connection.  yp  and  7?,  measure  the  average 
host-to-host  speed  of  Internet  connections  —  for  example,  the  rate  at  which  the  file  of  a  page  is  downloaded 
—  for  the  pairs  of  hosts  that  use  the  link. 

The  bit  rate  of  all  traffic  on  the  link  is  r  =  075.  Of  course,  t  <  (3  because  bits  cannot  be  put  on  the  link  at 
a  rate  faster  than  the  bandwidth.  A  larger  traffic  bit  rate  r  requires  a  larger  bandwidth  6.  Let  us  return  to 
the  path  across  the  Internet  for  the  Web  page  download  discussed  earlier.  Starting  from  the  link  connecting 
the  client  computer  to  the  Internet  and  proceeding  though  the  links,  r  tends  to  increase,  and  therefore,  so 
does  (3.  We  start  with  a  low  bandwidth  link,  say  1.5  megabits/sec,  then  to  a  link  at  the  edge  of  a  service 
provider  network,  say  156  megabits/sec,  and  then  to  the  core  links  of  the  provider,  say  10  gigabits/sec.  As  we 
continue  further,  we  move  from  the  core  to  the  service  provider  edge  to  a  link  connected  to  the  destination 
computer,  so  r  and  f3  tend  to  decrease. 


11.2.4  Queueing,  Best-Effort  Traffic,  and  QoS 

A  packet  arriving  for  transmission  on  a  link  is  presented  with  a  queueing  mechanism.  The  service  time  for  a 
packet  is  the  time  it  takes  to  put  the  packet  on  the  link,  which  is  the  packet  size  divided  by  the  bandwidth  (3 . 
If  there  are  any  packets  whose  transmission  is  not  completed,  then  the  packet  must  wait  until  these  packets 
arc  fully  transmitted  before  its  transmission  can  begin.  This  is  the  queueing  delay.  The  packets  waiting  for 
transmission  arc  stored  in  a  buffer,  a  region  in  the  memory  of  the  computer  or  router.  The  buffer  has  a  size. 
If  a  packet  arrives  and  the  buffer  is  full,  then  the  packet  is  dropped.  As  we  will  see,  the  arrival  process  for 
packets  on  a  link  is  long-range  dependent;  at  low  loads,  the  traffic  is  very  bursty,  but  as  the  load  increases, 
the  burstiness  dissipates.  For  a  fixed  r  and  (3,  bursty  traffic  results  in  a  much  larger  queue-height  distribution 
than  traffic  with  Poisson  arrivals. 

TCP,  the  predominant  protocol  for  managing  tile  transfers,  changes  the  rate  at  which  it  sends  packets  with 
file  contents.  TCP  increases  the  rate  when  all  goes  well,  but  reduces  the  rate  when  a  destination  computer 
indicates  that  a  packet  has  not  been  received;  the  assumption  is  that  congestion  somewhere  on  the  path  has 
led  to  a  buffer  overflow,  and  the  rate  reduction  is  needed  to  help  relieve  the  congestion.  In  other  words,  TCP 
is  closed-loop  because  there  is  feedback.  UDP  is  not  aware  of  dropped  packets  and  does  not  respond  to 
them. 

When  traffic  is  sent  across  the  Internet  using  TCP  or  UDP  and  this  queueing  mechanism,  with  no  attempts  to 
add  additional  protocol  features  to  improve  QoS,  then  the  traffic  is  referred  to  as  “best-effort”.  IP  networks 
are  a  best-effort  system  because  the  standard  protocols  make  an  effort  to  get  packets  to  their  destination,  but 
packets  can  be  delayed,  lost,  or  delivered  out  of  order.  Queueing  delay  and  packet  drops  degrade  the  QoS  of 
best-effort  traffic.  For  example,  for  Web  page  transfers,  the  result  is  a  longer  wait  by  the  user,  partly  because 
the  packets  sit  in  the  queue,  and  partly  because  of  TCP  reducing  its  sending  rate  when  retransmission  occurs. 
Best-effort  traffic  contrasts  with  priority  traffic,  which  when  it  arrives  at  a  router,  goes  in  front  of  best-effort 
packets.  Packets  for  voice  traffic  over  the  Internet  arc  often  given  priority. 
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11.3  The  Bandwidth  Estimation  Problem:  Formulation  and  Stream  Statistical  Properties 
11.3.1  Formulation 

Poor  QoS  resulting  from  delays  and  drops  on  an  Internet  link  can  be  improved  by  increasing  the  link  band¬ 
width  3.  The  service  time  decreases,  so  if  the  traffic  rate  r  remains  fixed,  the  queuing  delay  distribution 
decreases,  and  delay  and  loss  are  reduced.  Loss  and  delay  arc  also  affected  by  the  buffer  size;  the  larger 
the  buffer  size,  the  fewer  the  drops,  but  then  the  queueing  delay  has  the  potential  to  increase  because  the 
maximum  queueing  delay  is  the  buffer  size  divided  by  3. 

The  bandwidth  estimation  problem  is  to  choose  /3  to  satisfy  QoS  criteria.  The  resulting  value  of  3  is  the 
QoS  bandwidth.  The  QoS  utilization  is  the  value  of  u  =  r / f 3  corresponding  to  the  QoS  bandwidth.  When 
a  local  network,  such  as  a  company  or  university,  purchases  bandwidth  from  an  Internet  service  provider,  a 
decision  on  (3  must  be  made.  When  an  Internet  service  provider  designs  its  network,  it  must  choose  / 3  for 
each  of  its  links.  The  decision  must  be  based  on  the  traffic  load  and  QoS  criteria. 

Here,  we  address  the  bandwidth  estimation  problem  specifically  for  links  with  best-effort  traffic.  We  take 
the  QoS  criteria  to  be  delay  and  loss.  For  delay,  we  use  two  metrics:  a  delay,  5,  and  the  delay  probability, 
uj,  the  probability  that  a  packet  exceeds  the  delay.  For  loss,  we  suppose  that  the  decision  has  been  made 
to  choose  a  buffer  size  large  enough  that  drops  will  be  negligible.  This  is,  for  example,  consistent  with 
the  current  practice  of  service  providers  on  their  core  links  (Iyer,  Bhattacharyya,  Taft,  McKeown,  and  Diot 
2003).  Of  course,  a  large  buffer  size  allows  the  possibility  of  a  large  delay,  but  setting  QoS  values  for  5 
and  uj  allows  us  to  control  delay  probabilistically;  we  will  chose  uj  to  be  small.  The  alternative  is  to  use  the 
buffer  size  as  a  hard  limit  on  delay,  but  because  dropped  packets  are  an  extreme  remedy  with  more  serious 
degradations  of  QoS,  it  is  preferable  to  separate  loss  and  delay  control,  using  the  softer  probabilistic  control 
for  delay.  Stipulating  that  packet  loss  will  be  negligible  on  the  link  means  that  for  a  connection  that  uses  the 
link,  another  link  will  be  the  loss  bottleneck;  that  is,  if  packets  of  the  connection  are  dropped,  it  will  be  on 
another  link.  It  also  means  that  TCP  feedback  can  be  ignored  in  studying  the  bandwidth  estimation  problem. 


11.3.2  Packet  Stream  Statistical  Properties 

A  packet  stream  consists  of  a  sequence  of  arriving  packets,  each  with  a  size.  Let  v  be  the  arrival  number;  v 
=  1  is  the  first  packet,  v  =  2  is  the  second  packet,  and  so  forth.  Let  av  be  the  arrival  times,  let  tv  =  a„+\  —  av 
be  the  inter-arrival  times,  and  let  qv  be  the  size  of  the  packet  arriving  at  time  av .  The  statistical  properties  of 
the  packet  stream  will  be  described  by  the  statistical  properties  of  tv  and  qv  as  time  series  in  v. 

The  QoS  bandwidth  for  a  packet  stream  depends  critically  on  the  statistical  properties  of  the  tv  and  qv. 
Directly,  the  bandwidth  depends  on  the  queue-length  time  process.  But  the  queue-length  time  process 
depends  critically  on  the  stream  statistical  properties.  Here,  we  consider  best-effort  traffic.  It  has  persistent, 
long-range  dependent  tv  and  qv  (Ribeiro,  Riedi,  Crouse,  and  Baraniuk  1999;  Gao  and  Rubin  2001;  Cao, 
Cleveland,  Lin,  and  Sun  2001).  Persistent,  long-range  dependent  tv  and  qv  have  dramatically  larger  queue- 
size  distributions  than  those  for  independent  tv  and  qv  (Konstantopoulos  and  Lin  1996;  Erramilli,  Narayan, 
and  Willinger  1996;  Cao,  Cleveland,  Lin,  and  Sun  2001).  The  long-range  dependent  traffic  is  burstier  than 
the  independent  traffic,  so  the  QoS  utilization  is  smaller  because  more  headroom  is  needed  to  allow  for 
the  bursts.  This  finding  demonstrates  quite  clearly  the  impact  of  the  statistical  properties.  But  a  corollary 
of  the  finding  is  that  the  results  here  are  limited  to  best-effort  traffic  streams  (or  any  other  streams  with 
similar  statistical  properties).  Results  for  other  types  of  traffic  with  quite  different  statistical  properties  — 
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for  example,  links  carrying  voice  traffic  using  current  Internet  protocols  —  arc  different. 

Best-effort  traffic  is  not  homogeneous.  As  the  traffic  connection  load  c  increases,  the  arrivals  tend  toward 
Poisson  and  the  sizes  tend  tow  ard  indcpcndcntfCao,  Cleveland,  Lin,  and  Sun  2002;  Cao  and  Ramanan  2002). 
The  reason  is  the  increased  statistical  multiplexing  of  packets  from  different  connections;  the  intermingling 
of  the  packets  of  different  connections  is  a  randomization  process  that  breaks  down  the  correlation  of  the 
streams.  In  other  words,  the  long-range  dependence  dissipates.  This  means  that  in  our  bandwidth  estimation 
study,  we  can  expect  a  changing  estimation  mechanism  as  c  increases.  In  particular,  we  expect  “multiplexing 
gains”,  greater  utilization  due  to  the  reduction  in  dependence.  Because  of  the  change  in  properties  with  c, 
we  must  be  sure  to  study  streams  with  a  wide  range  of  values  of  c. 


11.4  FSD  Time  Series  Models  for  Packet  Arrivals  and  Sizes 

This  section  presents  FSD  time  series  models,  a  new  class  of  non-Gaussian,  long-range  dependent  mod¬ 
els  (Cao,  Cleveland,  Lin,  and  Sun  2002;  Cao,  Cleveland,  and  Sun  2004).  The  two  independent  packet-stream 
time  series  —  the  inter-arrivals,  tv,  and  the  sizes,  qv  —  are  each  modeled  by  an  FSD  model;  the  models  arc 
used  to  generate  synthetic  best-effort  traffic  streams  for  the  queueing  simulations  of  our  study. 

There  arc  a  number  of  known  properties  of  the  tr  and  qv  that  had  to  be  accommodated  by  the  FSD  models. 
First,  these  two  time  series  arc  long-range  dependent.  This  is  associated  with  the  important  discovery 
of  long-range  dependence  of  packet  arrival  counts  and  of  packet  byte  counts  in  successive  equal-length 
intervals  of  time,  such  as  10  ms  (Leland,  Taqqu,  Willinger,  and  Wilson  1994;  Paxson  and  Floyd  1995). 
Second,  the  tv  and  qv  arc  non-Gaussian.  The  complex  non-Gaussian  behavior  has  been  demonstrated  clearly 
in  important  work  showing  that  highly  nonlinear  multiplicative  multi-fractal  models  can  account  for  the 
statistical  properties  of  tv  and  qv  (Riedi,  Crouse,  Ribeiro,  and  Baraniuk  1999;  Gao  and  Rubin  2001).  These 
nonparametric  models  utilize  many  coefficients  and  a  complex  cascade  structure  to  explain  the  properties. 
Third,  the  statistical  properties  of  the  two  time  series  change  as  c  increases  (Cao,  Cleveland,  Lin,  and  Sun 
2002).  The  arrivals  tend  tow  ard  Poisson  and  the  sizes  tend  tow  ard  independent;  there  arc  always  long-range 
dependent  components  present  in  the  series,  but  the  contributions  of  the  components  to  the  variances  of  the 
series  goes  to  zero. 


11.4.1  Solving  the  Non-Gaussian  Challenge 

The  challenge  in  modeling  tv  and  qv  is  their  combined  non-Gaussian  and  long-range  dependent  properties, 
a  difficult  combination  that  does  not,  without  a  simplifying  approach,  allow  parsimonious  characterization. 
We  discovered  that  monotone  nonlinear  transformations  of  the  inter-arrivals  and  sizes  arc  very  well  fitted  by 
parsimonious  Gaussian  time  series,  a  very  simple  class  of  fractional  ARIMA  models  (Hosking  1981)  with 
a  small  number  of  parameters.  In  other  words,  the  transformations  and  the  Gaussian  models  account  for  the 
complex  multifractal  properties  of  the  tv  and  the  qv  in  a  simple  way. 


11.4.2  The  FSD  Model  Class 

Suppose  xv  for  v  =  1,2, ...  is  a  stationary  time  series  with  marginal  cumulative  distribution  function 
F(x;  <P)  where  0  is  a  vector  of  unknown  parameters.  Let  x*v  =  H ( xv ;  (P)  be  a  transformation  of  xv  such  that 
the  marginal  distribution  of  x*  is  normal  with  mean  0  and  variance  1.  We  have  H(xv;  (P)  =  G~  l(l7(.x;  (p)). 
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where  G(z)  is  the  cumulative  distribution  function  of  a  normal  random  variable  with  mean  0  and  variance 
1.  Next  we  suppose  x*  is  a  Gaussian  time  series  and  call  x*  the  “Gaussian  image”  of  xv. 

Suppose  x*  has  the  following  form: 

x*  =  Vl  -  6  sv  +  \f&  nv, 

where  sv  and  nv  arc  independent  of  one  another  and  each  has  mean  0  and  variance  1 .  nv  is  Gaussian  white 
noise,  that  is,  an  independent  time  series.  sv  is  a  Gaussian  fractional  ARIMA  (Hosking  1981) 

(I  —  B)dsv  =  ev  +  €u~i 

where  Bsv  =  su- i,  0  <  d  <  0.5,  and  ev  is  Gaussian  white  noise  with  mean  0  and  variance 

2_  (1-  d)V2(l-  d) 

2r(i  -  2d)  ' 


xv  is  &  fractional  sum- difference  ( FSD )  time  series.  Its  Gaussian  image,  x*,  has  two  components.  V 1  —  9sv 
is  the  long-range-dependent,  or  lrd,  component;  its  variance  is  1  —  0.  \[9  is  the  white-noise  component;  its 
variance  is  6. 


Let  px*  (/)  be  the  power  spectrum  of  the  x*.  Then 


Px * (/) 


d  -  t>w,  4cos2(t/)  , 

(4sin2(vr /)) 


+  0 


for  0  <  /  <  0.5.  As  /  — ►  0.5,  px*  (/)  decreases  monotonically  to  6.  As  /  — ►  0,  px*  (/)  goes  to  infinity  like 
sin  2,1  (Tif)  ~  /'  2d\  one  outcome  of  long-range  dependence.  For  nonnegative  integer  lags  k,  let  rx*(k), 
rs(k),  and  rn(k)  be  the  autocovariance  functions  of  x*,  sv,  and  nv,  respectively.  Because  the  three  series 
have  variance  1,  the  autocovariance  functions  arc  also  the  autocorrelation  functions.  rs(k)  is  positive  and 
falls  off  like  If'1  1  as  k  increases,  another  outcome  of  long-range  dependence.  For  k  >  0,  rn(k)  =  0  and 


rx*(k)  =  (1  -  9)rs(k). 


As  6  — >  1,  x*v  goes  to  white  noise:  px*  (/)  — >  1  and  rx*  ( k )  — ►  0  for  k  >  0.  The  change  in  the  autocovariance 
function  and  power  spectrum  arc  instructive.  As  9  gets  closer  to  1,  the  rise  of  px*  (/)  near  /  =  0  is  always 
to  order  /  2,1 ,  and  the  rate  of  decay  of  rx*{k )  for  large  k  is  always  k2'1  1 ;  but  the  ascent  of  px*{f)  at  the 
origin  begins  closer  and  closer  to  /  =  0  and  the  rx*  (/;:)  get  uniformly  smaller  by  the  multiplicative  factor 
1-9. 


11.4.3  Marginal  Distributions  of  qv  and  tv 

We  model  the  marginal  distribution  of  tv  by  a  Weibull  with  shape  A  and  scale  a,  a  family  with  two  unknown 
parameters.  Estimates  of  A  arc  almost  always  less  than  1.  The  Weibull  provides  an  excellent  approximation 
of  the  sample  marginal  distribution  of  the  t„  except  that  the  smallest  3%  to  5%  of  the  sample  distribution  is 
truncated  to  a  nearly  constant  value  due  to  certain  network  transmission  properties. 

The  marginal  distribution  of  qv  is  modeled  as  follows.  While  packets  less  than  40  bytes  can  occur,  it  is 
sufficiently  rare  that  we  ignore  this  and  suppose  40  <  qv  <  1500.  First,  we  provide  for  A  atoms  at  sizes 
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(s')  (s') 

(j>\  . . .  (pA  such  as  40  bytes,  512  bytes,  576  bytes,  and  1500  bytes  that  arc  commonly  occurring  sizes;  the 
atom  probabilities  arc  . . .  4$  •  For  the  remaining  sizes,  we  divided  the  interval  [40,  1500]  bytes  into 
C  sub-intervals  using  (7—1  distinct  break  points  with  values  that  arc  greater  than  40  bytes  and  less  than 
1500  bytes,  . . .  4^c-v  For  each  of  the  C  sub-intervals,  the  size  distribution  has  an  equal  probability 
for  the  remaining  sizes  (excluding  the  atoms)  in  the  sub-interval;  the  total  probabilities  for  the  sub-intervals 

(i) 

ar ,  (jrc  .  Typically,  with  just  3  atoms  at  40  bytes,  576  bytes,  and  1500  bytes,  and  with  just  2  break 
points  at  50  bytes  and  200  bytes,  we  get  an  excellent  approximation  of  the  marginal  distribution. 


11.4.4  Gaussian  Images  of  qv  and  tv 

The  transformed  time  series  t*  and  q*  appeal-  to  be  quite  close  to  Gaussian  processes.  Some  small  amount 
of  non-Gaussian  behavior  is  still  present  but  it  is  minor.  The  autocorrelation  structure  of  these  Gaussian 
images  are  very  well  fitted  by  the  FSD  autocorrelation  structure. 

The  parameters  of  the  FSD  model  are  the  following: 

•  qv  marginal  distribution:  A  atom  probabilities  of  '  at  A  sizes  (7—1  break  points  cpf'1  and  (7 

(i) 

sub-interval  probabilities  (f)\ 

•  tv  marginal  distribution:  shape  A  and  scale  a 

•  q*  time  dependence:  fractional  difference  coefficient  d<q>  and  white-noise  variance  0(q) 

•  t*  time  dependence:  fractional  difference  coefficient  d<L>  and  white-noise  variance  6»W. 

We  found  that  the  d}q>  and  ri'F  do  not  depend  on  c;  this  is  based  on  empirical  study,  and  it  is  supported 
by  theory.  The  estimated  values  are  0.410  and  0.411,  respectively.  We  take  the  value  of  each  of  these  two 
parameters  to  be  0.41.  We  found  that  as  c  increases,  estimates  of  A,  0iqK  and  0it}  all  tend  toward  1.  This 
means  the  tv  tend  to  a  independent  exponentials,  a  Poisson  process  and  the  qv  tend  toward  independence.  In 
other  words,  the  statistical  models  account  for  the  change  in  the  tv  and  qv  and  c  increases  that  was  discussed 
earlier.  We  estimated  these  three  parameters  and  a  by  partial  likelihood  methods  with  d<q>  and  d(lqi  fixed  to 
0.41.  The  marginal  distribution  of  qv  on  a  given  link  does  not  change  with  c,  but  it  does  change  from  link 
to  link.  To  generate  traffic,  we  must  specify  the  atom  and  sub-interval  probabilities.  This  provides  a  mean 
packet  size  /j(qy  measured  in  bits/packet. 


11.5  Packet  Stream  Data:  Live  and  Synthetic 

We  use  packet  stream  data,  values  of  packet  arrivals  and  sizes,  in  studying  the  bandwidth  estimation  problem. 
They  are  used  as  input  traffic  for  queueing  simulations.  There  are  two  types  of  streams:  live  and  synthetic. 
The  live  streams  are  from  packet  traces,  data  collection  from  live  Internet  links.  The  synthetic  streams  are 
generated  by  the  FSD  models. 


11.5.1  Live  Packet  Streams 

A  commonly-used  measurement  framework  for  empirical  Internet  studies  results  in  packet  traces  (Claffy, 
Braun,  andPolyzos  1995;  Paxson  1997;  Duffield,  Feldmann,  Friedmann,  Greenberg,  Greer,  Johnson,  Kalmanek, 
B.Krishnamurthy,  Lavelle,  Mishra,  Ramakrishnan,  Rexford,  True,  ,  and  van  der  Merwe  2000).  The  arrival 
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time  of  each  packet  on  a  link  is  recorded  and  the  contents  of  the  headers  arc  captured.  The  vast  majority  of 
packets  arc  transported  by  TCP,  so  this  means  most  headers  have  40  bytes,  20  for  TCP  and  20  for  IP.  The  live 
packet  traffic  is  measured  by  this  mechanism  over  an  interval;  the  time-stamps  provide  the  live  inter-arrival 
times,  tv,  and  the  headers  contain  information  that  provides  the  live  sizes,  qv ;  so  for  each  trace,  there  is  a 
stream  of  live  arrivals  and  sizes. 

The  live  stream  database  used  in  this  presentation  consists  of  349  streams,  90  sec  or  5  min  in  duration,  from 
6  Internet  links  that  we  name  BELL,  NZIX,  AIX1,  AIX2,  MLN1,  and  MLN2.  The  measured  streams  have 
negligible  delay  on  the  link  input  router.  The  mean  number  of  simultaneous  active  connections,  c,  ranges 
from  49  connections  to  18,976  connections.  The  traffic  bit  rate,  r,  ranges  from  1.00  megabits/sec  to  348 
megabits/sec. 

BELL  is  a  100  megabit/sec  link  in  Murray  Hill,  NJ,  USA  connecting  a  Bell  Labs  local  network  of  about 
3000  hosts  to  the  rest  of  the  Internet.  The  transmission  is  half-duplex,  so  both  directions,  in  and  out,  are 
multiplexed  and  carried  on  the  same  link,  and  a  stream  is  the  multiplexing  of  both  directions;  but  to  keep  the 
variable  c  commensurate  for  all  of  the  six  links,  the  two  directions  for  each  connection  is  counted  as  two.  In 
this  presentation  we  use  195  BELL  traces,  each  5  min  in  length.  NZIX  is  the  100  megabit/sec  New  Zealand 
Internet  exchange  hosted  by  the  ITS  department  at  the  University  of  Waikato,  Hamilton,  New  Zealand,  that 
served  as  a  peering  point  among  a  number  of  major  New  Zealand  Internet  Service  Providers  at  the  time  of 
data  collection  (http://wand.es. waikato. ac.nz/wand/wits/nzix/2/ ).  All  arriving  packets  from  the  input/output 
ports  on  the  switch  arc  mirrored,  multiplexed,  and  sent  to  a  port  where  they  arc  measured.  Because  all 
connections  have  two  directions  at  the  exchange,  each  connection  counts  2  as  for  BELL.  In  this  presentation 
we  use  84  NZIX  traces,  each  5  min  in  length.  AIX1  and  AIX2  arc  two  separate  622  megabit/sec  OC12 
packet-over-sonet  links,  each  carrying  one  direction  of  the  traffic  between  NASA  Ames  and  the  MAE-West 
Internet  exchange.  In  this  presentation  we  use  23  AIX1  and  23  AIX2  traces,  each  90  sec  in  length.  AIX1  and 
AIX2  streams  were  collected  as  paid  of  a  project  at  the  National  Laboratory  for  Applied  Network  Research 
where  the  data  arc  collected  in  blocks  of  90  sec  (http://pma.nlanr.net/PMA  ).  MLN1  and  MLN2  arc  two 
separate  2.5  gigabit/sec  OC48  packet-over-sonet  links  on  the  network  of  the  service  provider  Metropolitan 
Liber  Network;  each  link  carries  one  direction  of  traffic  between  San  Jose,  California,  USA  and  Seattle, 
Washington,  USA.  In  this  presentation  we  use  12  MLN1  and  12  MLN2  traces,  each  5  min  in  length. 

The  statistical  properties  of  streams,  as  we  have  stated,  depend  on  the  connection  load  c,  so  it  is  important 
that  the  time  interval  of  a  live  stream  be  small  enough  that  c  does  not  vary  appreciably  over  the  interval.  Lor 
any  link,  there  is  diurnal  variation,  a  change  in  c  with  the  time  of  day  due  to  changes  in  the  number  of  users. 
We  chose  5  minutes  to  be  the  upper  bound  of  the  length  of  each  stream  to  ensure  stationarity.  BELL,  NXIZ, 
MLN1,  and  MLN2  streams  arc  5  min.  AIX1  and  AIX2  traces  arc  90  sec  because  the  sampling  plan  at  these 
sites  consisted  of  noncontiguous  90-sec  intervals. 


11.5.2  Synthetic  Packet  Streams 

The  synthetic  streams  arc  arrivals  and  sizes  generated  by  the  PSD  models  for  tv  and  qv.  Each  of  the  live 
streams  is  fitted  by  two  PSD  models,  one  for  the  tv  and  one  for  the  qv,  and  a  synthetic  stream  of  5  min  is 
generated  by  the  models.  The  generated  tv  are  independent  of  the  generated  qv,  which  is  what  we  found  in 
the  live  data.  The  result  is  349  synthetic  streams  that  match  the  statistical  properties  collectively  of  the  live 
streams. 


22 


11.6  Queueing  Simulation 


We  study  the  bandwidth  estimation  problem  through  queueing  simulation  with  an  infinite  buffer  and  a  first- 
in-first-out  queueing  discipline.  The  inputs  to  the  queues  arc  the  arrivals  and  sizes  of  the  349  live  and  349 
synthetic  packet  streams  described  in  Section  11.5. 

For  each  live  or  synthetic  stream,  we  carry  out  25  runs,  each  with  a  number  of  simulations.  For  each  run 
we  pick  a  delay  S  and  a  delay  probability  u>;  simulations  are  carried  out  to  find  the  QoS  bandwidth  j3,  the 
bandwidth  that  results  in  delay  probability  to  for  the  delay  6.  This  also  yields  a  QoS  utilization  u  =  r  /  3. 
We  use  5  delays  —  0.001  sec,  0.005  sec,  0.010  sec,  0.050  sec,  0.100  sec  —  and  5  delay  probabilities  — 
0.001,  0.005,  0.01,  0.02,  and  0.05  —  employing  all  25  combinations  of  the  two  delay  criteria.  For  each 
simulation  of  a  collection,  5  is  fixed  a  priori.  We  measure  the  queueing  delay  at  the  arrival  times  of  the 
packets,  which  determines  the  simulated  queueing  delay  process.  From  the  simulated  process  we  find  the 
delay  probability  for  the  chosen  <5.  We  repeat  the  simulation,  changing  the  trial  QoS  bandwidth,  until  the 
attained  delay  probability  approximately  matches  the  chosen  delay  probability  ui.  The  optimization  is  easy 
because  u>  decreases  as  the  trial  QoS  bandwidth  increases  for  fixed  5. 

In  the  optimization,  we  do  not  allow  the  utilization  to  go  above  0.97;  in  other  words,  if  the  true  QoS  uti¬ 
lization  is  above  0.97,  we  set  it  to  0.97.  The  reason  is  that  we  use  the  logit  scale,  log(rt/(l  —  u)),  in  the 
modeling;  above  about  0.97,  the  scale  becomes  very  sensitive  to  model  mis-specification  and  the  accuracy 
of  the  simulation,  even  though  the  utilizations  above  0.97  for  practical  purposes  are  nearly  equal.  Similarly, 
we  limit  the  lower  range  of  the  utilizations  to  be  0.05. 

The  result  of  the  25  runs  for  each  of  the  349  live  and  349  synthetic  streams  is  25  measurements,  one  per  run, 
of  each  of  five  variables:  QoS  utilization  u,  delay  S,  delay  probability  t o,  mean  number  of  active  connections 
c,  and  bit  rate  r.  The  first  three  variables  vary  from  run  to  run;  the  last  two  variables  arc  the  same  for  the  25 
runs  for  a  stream  because  they  measure  the  stream  statistical  properties.  By  design,  the  range  of  5  is  0.001 
sec  to  0.100  sec,  and  the  range  of  ui  is  0.001  to  0.05.  The  range  of  the  QoS  utilizations  is  0.05  to  0.97.  The 
two  additional  variables  r  and  c,  which  measure  the  statistical  properties  of  the  streams,  are  constant  across 
the  25  runs  for  each  stream,  c  ranges  from  49  connections  to  18,976  connections,  and  r  ranges  from  1.00 
megabits/sec  to  348  megabits/sec. 


11.7  Model  Building:  A  Bandwidth  Formula  Plus  Random  Error 

This  section  describes  the  process  of  building  the  Best-Effort  Delay  Model,  which  is  the  Best-Effort  Delay 
Formula  plus  random  error.  The  model  describes  the  dependence  of  the  utilization  u  on  the  delay  5,  the 
delay  probability  ui,  the  traffic  bit  rate  r,  and  the  expected  number  of  active  connections  c.  The  modeling 
process  involves  both  theory  and  empirical  study,  and  establishes  a  basis  for  the  model. 

The  theoretical  basis  is  queueing  theory.  The  empirical  basis  is  the  delay  data  from  the  queueing  simulations, 
the  measurements  of  the  five  variables  described  in  Section  11.6.  The  following  notation  is  used  for  the 
values  of  these  five  variables  for  either  the  live  delay  data  or  the  synthetic  delay  data.  5j,  for  j  =  1  to  5,  are 
the  5  values  of  the  delay  in  increasing  order,  and  tok,  for  k  =  1  to  5,  are  the  5  values  of  the  delay  probability 
in  increasing  order.  uKlk  is  the  QoS  utilization  for  delay  Sj,  delay  probability  u>k,  and  stream  i,  where  i  =  1 
to  349.  t i  is  the  traffic  bit  rate  and  ct  is  the  mean  number  of  active  connections,  both  for  stream  i. 


23 


11.7.1  Strategy:  Initial  Modeling  of  Dependence  on  5  and  u 


The  structure  of  the  data  provides  an  opportunity  for  careful  initial  study  of  the  dependency  of  the  uVJ}.  on  Sj 
and  ujk-  We  have  25  measurements  of  each  of  these  variables  for  each  stream  i,  and  for  these  measurements, 
both  t i  and  c,  arc  constant.  We  start  our  model  building  by  exploiting  this  opportunity. 

We  will  consider  modeling  each  stream  separately,  but  hope  there  will  be  model  consistency  across  streams 
that  allows  simplification.  If  such  simplicity  is  to  occur,  it  is  likely  that  it  will  require  a  monotone  trans¬ 
formation  of  the  Uijk  because  they  vary  between  0  and  1 .  So  we  begin,  conceptually,  with  a  model  of  the 
form 

f('Uijk')  —  9i{^j 4-  e ijk ■ 

The  eijk  are  a  sample  from  a  distribution  with  mean  0.  /  is  a  monotone  function  of  u,  and  gt  is  a  function 
of  <5  and  a;  we  want  to  choose  /  to  make  g%  as  simple  as  possible,  that  is,  vary  as  little  as  possible  with  i. 


11.7.2  Conditional  Dependence  of  u  on  5 

We  start  our  exploration  of  the  data  by  taking  f(u)  =  u  and  suppose  that  a  logical  scale  for  S  is  the  log.  In 
all  cases,  we  use  log  base  2  and  indicate  this  by  writing  log2  in  our  formulas.  We  do  not  necessarily  believe 
that  this  identity  function  for  /  is  the  right  transformation,  but  it  is  helpful  to  study  the  data  initially  on  the 
untransformed  utilization  scale. 

Our  first  step  is  to  explore  the  conditional  dependence  of  uyjk  on  log2(Sj)  given  ay.  and  the  stream  i  by 
trellis  display  (Becker,  Cleveland,  and  Shyu  1996).  For  each  combination  of  the  delay  probability  a /.  and 
the  stream  i,  we  graph  ul;jk  against  log2(bj).  We  did  this  once  for  all  349  live  streams  and  once  for  all  349 
synthetic  streams.  Figure  1  illustrates  this  by  a  trellis  display  for  16  of  the  live  streams.  The  16  streams  were 
chosen  to  nearly  cover  the  range  of  values  of  the  t,.  Let  Trv\,  for  v  =  I  to  349,  be  the  values  ordered  from 
smallest  to  largest,  and  take  Trv\  to  be  the  quantile  of  the  empirical  distribution  of  the  values  of  order  v / 349. 
Then  we  chose  the  16  streams  whose  ranks  v  yield  orders  the  closest  to  the  16  equally-spaced  orders  from 
0.05  to  0.95.  On  the  figure,  there  arc  80  panels  divided  into  10  columns  and  8  rows.  On  each  panel,  uyjk  is 
graphed  against  log2(4jj  for  one  value  of  a/,,  and  one  stream;  the  strip  labels  at  the  top  of  each  panel  give 
the  value  of  uy .  and  the  rank  of  the  stream.  There  arc  5  points  per  panel,  one  for  each  value  of  log2(()?). 

Figure  1  shows  a  number  of  overall  effects  of  r,  5,  and  ui  on  u.  For  each  pair  of  values  of  a  and  r,  there  is 
an  increase  in  u  with  6,  a  strong  main  effect  in  the  data.  In  addition,  there  is  an  increase  with  r  for  fixed  6 
and  a,  another  strong  main  effect.  There  is  also  a  main  effect  for  a,  but  smaller  in  magnitude  than  for  the 
other  two  variables.  The  dependence  of  u  on  log2(b)  is  nonlinear,  and  changes  substantially  with  the  value 
of  r;  as  r  increases,  the  overall  slope  in  u  as  function  of  log2(<5)  first  increases  and  then  decreases.  In  other 
words,  there  is  an  interaction  between  log2(<5)  and  r.  Such  an  interaction  complicates  the  dependence,  so 
we  search  further  for  a  transformation  f  of  u  that  removes  the  interaction.  This  pattern  occurs  when  all  of 
the  live  streams  or  all  of  the  synthetic  streams  arc  plotted  in  the  same  way. 

There  is  an  interaction  between  log2(<5)  and  r  in  paid  because  once  u  is  close  to  1,  where  there  is  little 
room  for  change  as  a  function  of  log2(<5).  For  this  reason,  we  tried  expanding  the  scale  at  1  by  taking  the 
function  f(u)  =  log2(l  —  u).  This  did  not  achieve  appreciably  greater  simplicity  because  nonlinearity  and 
an  interaction  arc  still  strongly  present,  but  the  interaction  cause  is  behavior  for  smaller  values  of  r. 

The  nature  of  the  remaining  interaction  for  f(  u)  =  log2(l  —  u)  suggests  that  a  logit  transformation  might 
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do  better: 


f(u)  =  logit2  (it)  =  log2 


u 


1  —  u 


Figure  2  plots  logit2(rii:)fc)  against  log2((i?)  using  the  same  streams  and  method  as  Figure  1.  The  logit 
function  greatly  simplifies  the  dependence.  The  dependence  on  log2(d)  is  linear.  There  does  not  appear  to 
be  any  remaining  interaction  among  the  three  variables:  log2(<5),  r,  and  uj.  To  help  show  this,  16  lines  with 
different  intercepts  but  the  same  linear  coefficient,  have  been  drawn  on  the  panels.  The  method  of  fitting 
will  be  described  shortly.  The  lines  provide  an  excellent  fit. 


11.7.3  Theory:  The  Classical  Erlang  Delay  Formula 


The  packet  arrivals  aj  are  not  Poisson,  although  they  to  tend  toward  Poisson  as  c  and  r  increase.  The  packet 
sizes,  and  therefore  the  service  times,  are  not  independent  exponential;  the  packet  sizes  have  a  bounded 
discrete  distribution  and  arc  long-range  dependent,  although  they  tend  to  independent  as  c  and  r  increase. 
Still,  we  will  use,  as  a  suggestive  case,  the  results  for  Poisson  arrivals  and  i.i.d.  exponential  service  times 
to  provide  guidance  for  our  model  building.  Erlang  showed  that  for  such  a  model,  the  following  equation 
holds  (Cooper  1972): 

uj  =  ue~{1~u)l35. 

Substituting  for  (3  =  t/u  and  taking  the  negative  log  of  both  sides  we  have 


-log2(w) 


log2(«)  +  log2(e) 


1  —  u 


St. 


u 


(1) 


Because  uj,  which  ranges  from  0.001  to  0.05,  is  small  in  the  majority  of  our  simulations  compared  with  u, 
we  have,  approximately, 

1  —  ?/ 

-log2M  =  log2(e)  'St. 

u 

Taking  logs  of  both  sides  and  rearranging  we  have 


logit2 (u)  =  log2(log2(e))  +  log2(r)  +  log2(<5)  -  log2(-  log2(u;)).  (2) 


So  certain  aspects  of  the  simplicity  of  this  classical  Erlang  delay  formula  occur  also  in  the  pattern  for  our 
much  more  statistically  complex  packet  streams.  In  both  cases,  logit2(n)  is  additive  in  functions  of  r,  6,  and 
c u,  and  the  dependence  is  linear  in  log2(<5). 


11.7.4  Conditional  Dependence  of  u  on  uj 

The  approximate  Erlang  delay  formula  suggests  that  we  tty  the  term  —  log2(—  log2(cu)),  the  negative  com¬ 
plementary  log  of  uj,  in  the  model.  In  addition,  as  we  will  see  in  Section  11.9,  certain  asymptotic  results 
suggest  this  term  as  well.  We  studied  the  dependence  of  logtt^rr)  on  —  log2(—  log2(l  —  uj))  for  all  synthetic 
and  live  streams  using  trellis  display  in  the  same  way  that  we  studied  the  dependence  on  log2(<5).  Figure  3 
is  a  trellis  plot  using  the  same  16  live  streams  as  in  Figure  2.  On  each  panel,  logit 2(uijk)  is  graphed  against 
—  log2(—  log2(a;fc))  for  one  value  of  Sj  and  one  stream. 

Figure  3  shows  that  the  guidance  from  the  Erlang  formula  is  on  target;  logit2  (u)  is  linear  in  —  log2  (—  log2  (w)), 
and  the  slope  remains  constant  across  streams  and  across  different  values  of  S.  To  help  show  this,  lines  with 
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Logit  Utilization  (logit  base  2  fraction) 


Logit  Utilization  (logit  base  2  fraction) 


-Log(-Log(Probability)  (-complementary  log  base  2  fraction) 


Figure  3:  Logit  utilization,  logit2(u),  is  graphed  against  the  negative  complementary  log  of  the  delay  prob 
ability,  —  log2(—  log2(cu)),  given  the  delay,  5,  and  the  stream,  i. 


the  same  linear  coefficient  but  different  intercepts,  have  been  drawn  on  the  panels.  The  lines  provide  an 
excellent  fit  except  for  the  errant  points  for  high  utilizations  observed  earlier.  The  method  of  fitting  will  be 
described  shortly.  This  pattern  occurs  when  all  of  the  live  streams  or  all  of  the  synthetic  streams  arc  plotted 
in  the  same  way. 


A  Stream-Coefficient  Delay  Model 

The  empirical  findings  in  Figures  2  and  3  and  the  guidance  form  the  Erlang  Delay  Formula  have  led  to  a 
very  simple  model  that  tits  the  data: 

,  logit 2(uijk)  =  Hi  +  os  log 2{5j)  +  ou(-  log2 (  log2(wfc)))  +  eijk  (3) 

where  the  etjk  arc  realizations  of  an  error  random  variable  with  mean  0  and  median  absolute  deviation  m(e). 
The  fM  are  stream  coefficients;  they  change  with  the  packet  stream  i  and  characterize  the  statistical  properties 
of  the  stream. 

We  fitted  the  stream-coefficient  delay  model  of  Equation  3  twice,  once  to  the  349  live  streams  and  once  to  the 
349  synthetic  streams;  in  other  words,  we  estimated  the  coefficients  Hi ,  os,  and  ou  twice.  Data  exploration 
suggests  that  the  error  distribution  has  longer  tails  than  the  normal,  so  we  used  the  hi  square  method  of  robust 
estimation  (Mosteller  and  Tukey  1977).  The  estimates  of  os  and  ou  arc 

Live:  os  =  0.411  6W  =  0.868 
Synthetic:  ds  =  0.436  <5^  =  0.907. 

The  two  sets  of  estimates  are  very  close  in  the  sense  that  the  fitted  equation  is  very  close,  that  is,  results  in 
very  similar  fitted  QoS  utilizations.  For  the  16  streams  shown  in  Figures  2  and  3,  the  lines  are  drawn  using 
the  formula  in  Equation  3  with  the  hi  square  parameter  estimates  —  ds,  ou,  and  //,. 

Because  of  the  long-tailed  error  distribution,  we  use  the  median  absolute  deviation,  me,  as  a  measure  of  the 
spread.  The  estimates  from  the  residuals  of  the  two  fits  arc 

Live:  m(e)  =  0.210 
Synthetic:  m(e)  =  0.187. 

The  estimates  arc  very  small  compared  with  the  variation  in  logit2(u).  In  other  words,  the  stream-coefficient 
delay  model  provides  a  very  close  fit  to  the  logit2(rq Of  course,  this  was  evident  from  Figures  2  and  3 
because  the  fitted  lines  are  quite  close  to  the  data. 


11.7.5  Strategy:  Incorporating  Dependence  on  r  and  c  for  Practical  Estimation 

The  coefficient  /q  in  the  stream-coefficient  delay  model  of  Equation  3  varies  with  the  packet  stream  and 
reflects  how  the  changing  statistical  properties  of  the  streams  affect  the  QoS  utilization.  Part  of  the  simplicity 
of  the  model  is  that  a  single  number  characterizes  how  the  statistical  properties  of  a  stream  affect  the  QoS 
bandwidth.  However,  the  model  cannot  be  used  as  a  practical  matter  for  bandwidth  estimation  because  it 
requires  a  value  of  h,  which  would  not  typically  be  known.  If  we  knew  details  of  the  traffic  characteristics 
in  detail  for  the  link,  for  example,  if  we  had  FSD  parameters,  we  could  generate  traffic  and  run  simulations 
to  determine  h  and  therefore  the  bandwidth.  This  might  be  possible  in  certain  cases,  but  in  general  is  not 
feasible. 
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What  we  must  do  is  start  with  Equation  3,  and  find  readily-available  variables  that  measure  stream  statistical 
properties  and  can  replace  //  in  the  stream-coefficient  delay  model.  We  carry  out  this  task  in  the  remainder 
of  this  section.  Two  variables  replace  //,  the  bit  rate  t  and  the  mean  number  of  active  connections  c,  with 
their  values  of  t,  and  c,  for  each  of  our  packet  streams.  We  will  use  both  theory  and  empirical  study,  as  we 
did  for  the  stream-coefficient  delay  model,  to  carry  out  the  model  building. 


11.7.6  Theory:  Fast-Forward  Invariance,  Rate  Gains,  and  Multiplexing  Gains 

Figures  2  and  3  show  that  the  QoS  utilization  u  increases  with  r.  There  are  two  causes:  rate  gains  and 
multiplexing  gains.  Because  c  is  positively  correlated  with  r,  u  increases  with  c  as  well.  But  r  and  c 
measure  different  aspects  of  the  load,  which  is  important  to  the  modeling,  r  is  equal  to  075  where  y&  is  the 
connection  bit  rate  in  bits/sec/connection.  y&  measures  the  end-to-end  speed  of  transfers,  c  measures  the 
amount  of  multiplexing.  An  increase  in  either  increases  r. 

First  we  introduce  fast-forwarding.  Consider  a  generalized  packet  stream  with  bit  rate  r  input  to  a  queue 
without  any  assumptions  about  the  statistical  properties.  The  packet  sizes  can  be  any  sequence  of  positive 
random  variables,  and  the  inter-arrivals  can  be  any  point  process.  Suppose  we  arc  operating  at  the  QoS 
utilization  u  =  t/ ft  for  QoS  delay  criteria  8  and  to.  Now  for  h  >  1,  we  speed  up  the  traffic  by  dividing  all 
inter-arrival  times  tv  by  h.  The  packet  stream  has  a  rate  change:  the  statistical  properties  of  the  tv  change 
only  by  a  multiplicative  constant.  A  rate  increase  of  h  increases  75  by  the  factor  h  but  not  c.  The  bit  rate 
r  changes  to  hr.  Suppose  we  also  multiply  the  bandwidth  ft  by  h,  so  that  the  utilization  u  is  constant. 
Then  the  delay  process  of  the  rate-changed  packet  stream  is  the  delay  process  for  the  original  packet  stream 
divided  by  h.  That  is,  if  we  carried  out  a  simulation  with  a  live  or  synthetic  packet  stream,  and  then  repeated 
the  simulation  with  the  rate  change,  then  the  delay  of  each  packet  in  the  second  simulation  would  be  the 
delay  in  the  first  divided  by  h.  The  traffic  bit  rate,  the  bandwidth,  and  the  delay  process  are  speeded  up 
by  the  factor  h,  but  the  variation  of  the  packet  stream  and  the  queueing  otherwise  remain  the  same.  If  we 
changed  our  delay  criterion  from  8  to  8/h,  then  the  QoS  utilization  u  would  be  the  same,  which  means  the 
QoS  bandwidth  is  hft.  It  is  as  if  we  videotaped  the  queueing  mechanism  in  the  first  simulation  and  then 
produced  the  second  by  watching  the  tape  on  fast  forward  with  the  clock  on  the  tape  player  running  faster 
by  the  factor  h  as  well;  we  will  call  this  phenomenon  fast-forward  invariance. 

Fet  us  now  reduce  some  of  the  speed-up  of  the  fast  forwarding.  We  will  divide  the  tv  by  h  which  increases 
7/,  by  the  factor  h.  But  we  hold  8  fixed  and  do  not  decrease  by  the  factor  l/h.  What  is  the  new  QoS  u  that 
satisfies  the  delay  criteria  8  and  cc?  Since  u  satisfies  the  criteria  for  delay  8/h,  we  have  room  for  more  delay, 
so  u  can  increase.  In  other  words,  a  rate  increase  results  in  utilization  gains  for  the  same  8.  This  is  the  rate 
gain. 

Now  suppose  we  hold  r  fixed  but  increase  c  by  the  factor  h  >  1.  This  means  that  7/,  must  be  reduced  by  the 
factor  l/h.  Now  the  statistical  properties  change  in  other  ways  due  to  the  increased  multiplexing.  As  we 
have  seen  in  Section  1 1.4,  the  tv  tend  toward  Poisson  and  the  qv  tend  toward  independence.  The  dissipation 
of  the  long-range  dependence  of  the  packet  streams,  as  well  as  the  tendency  of  the  marginal  distribution 
of  tv  toward  exponential,  tends  to  decrease  the  queueing  delay  distribution  and  thereby  increase  the  QoS 
utilization.  In  other  words,  there  are  statistical  multiplexing  gains. 

These  theoretical  considerations  lead  us  to  two  important  conclusions  about  modeling.  First,  we  want  to  be 
sure  that  whatever  model  results,  it  must  obey  the  principle  of  fast-forward  invariance.  Second,  it  is  unlikely 
that  it  will  be  enough  to  model  with  just  r.  If  75  were  constant  across  the  Internet,  r  and  c  would  measure 
exactly  the  same  thing  for  our  purposes  and  we  would  have  no  need  for  c  beyond  r.  But  if  7*,  changes 
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substantially,  as  seems  likely,  then  we  will  need  c  as  well.  Figure  4  shows  the  6  medians  from  the  349  values 
of  log2(7b)  for  our  live  streams  broken  up  into  6  groups  by  the  link.  The  range  of  the  medians  is  about  4  log 
base  2  bits/sec/connection,  which  means  that  the  medians  of  7 &  change  by  a  factor  of  16.  One  link,  NZIX, 
is  appreciably  slower  than  the  others. 


Log  Connection  Bitrate  (log  base  2  bits/sec/connection) 

Figure  4:  Dotplot  of  median  log  connection  bit  rate,  log2(7&),  for  6  Internet  links. 


11.7.7  Modeling  with  r  and  c 

We  will  begin  by  modeling  just  with  r  to  see  if  this  can  explain  the  observed  utilizations  without  c.  The 
approximate  Erlang  delay  formula  in  Equation  2  suggests  that  the  dependence  of  the  stream  coefficients  on 
r  is  lineal-  in  log2(r).  This  means  the  model  for  logit 2(uijk)  is 

logit2(uyfc)  =  o  +  oT  log2(r?;)  +  o, 5  log2(<5,-)  +  o ><„(-  log2(-  log2(u;fc)))  +  ipijk-  (4) 

where  the  are  realizations  of  an  error  random  variable  with  mean  0.  In  our  initial  explorations  for  the  tit 
and  the  residuals  we  discovered  that  the  spread  of  the  residuals  increased  with  increasing  Sj.  So  we  model 
the  median  absolute  deviation  of  the  ipijk  by  m^j  (ip),  allowing  it  to  change  with  Sj. 

We  fitted  the  model  to  the  live  data  and  to  the  synthetic  data  using  bisquare,  also  accommodating  the  chang¬ 
ing  value  of  msj  (ip).  Figure  5  shows  dotplots  of  the  6  medians  of  the  residuals  for  the  6  links.  There  is  a 
clear  link  effect,  mimicing  the  behavior  Figure  4;  the  two  links  with  the  largest  and  smallest  residual  medi¬ 
ans  are  the  two  with  the  largest  and  smallest  median  connection  bit  rates.  The  behavior  of  these  extremes 
is  what  we  would  expect.  For  example,  NZIX  has  the  smallest  median  7^,  so  its  bit  rate  under-predicts  the 
utilization  because  a  stream  at  NZIX  with  a  certain  t  has  more  multiplexing  than  average  than  streams  at 
other  links  with  the  same  r,  which  means  the  favorable  statistical  properties  push  the  utilization  higher  than 
expected  under  the  model.  The  same  plot  for  the  synthetic  streams  shows  the  same  effect. 

In  addition,  there  is  another  inadequacy  of  this  first  model.  Because  7 5  is  changing,  we  want  the  model 
to  obey  the  principle  of  fast-forward  invariance,  but  it  does  not  because  the  estimates  of  oT  and  05  are  not 
equal. 

We  will  enlarge  the  bandwidth  model  by  adding  the  variable  log2(c).  Because  log2(r)  is  used  in  the  initial 
model,  adding  log2(c)  is  equivalent  to  adding  75.  In  doing  this  we  want  an  equation  that  obeys  fast-forward 


31 


Median  Residual  Logit  Utilization  (logit  base  2  fraction) 


Figure  5 :  Dotplot  of  link  median  residuals  from  the  first  fit  to  the  logit  utilization  using  only  the  bit  rate  r  to 
characterize  stream  statistical  properties. 


invariance:  if  we  hold  c  fixed,  multiply  r  by  h,  and  divide  5  by  h,  then  we  do  not  want  a  change  in  the  QoS 
utilization.  This  is  achieved  by  the  Best-Effort  Delay  Model: 

logit  2(uijk)  =  o  +  oclog2(cj)  +  oTSlog2(Ti5j)  +  oaJ(-log2(-log2(cjfc)))  +ipijk  (5) 

where  the  ip^  arc  error  variables  with  mean  0  and  median  absolute  deviation  mgj  (ip).  Fast- forward  invari¬ 
ance  is  achieved  by  entering  r  and  5  as  a  product. 

We  fitted  the  enlarged  of  Equation  5  to  the  live  streams  and  to  the  synthetic  streams  using  bisquare  because 
our  exploration  showed  that  the  error  distribution  has  longer  tails  than  the  normal;  the  estimation  included  a 
normalization  to  adjust  for  the  m^.  (ip).  The  bisquare  estimates  of  o,  oc,  org,  and  oUJ  are 

Live:  o  =  —8.933  oc  =  0.420  oTs  =  0.444  ou  =  0.893 

Synthetic:  6  =  —8.227  dc  =  0.353  oTg  =  0.457  bu  =  0.952. 

The  two  sets  of  estimates  are  very  close  in  the  sense  that  the  fitted  equations  arc  close.  The  estimates  of 
mgj  (ip),  the  median  absolute  deviations,  rhgj  (ip),  of  the  residuals,  arc 

Delay  :  1  ms  5  ms  10  ms  50  ms  100  ms 

Live:  0.211  0.312  0.372  0.406  0.484 

Synthetic:  0.169  0.322  0.356  0.380  0.457 

Again,  the  two  sets  of  estimates  arc  close. 

It  is  important  to  consider  whether  the  added  variable  c  is  contributing  in  a  significant  way  to  the  variability 
in  logit 2(uijk),  and  does  not  depend  fully  on  the  single  link  NZIX.  We  used  the  partial  standardized  residual 
plot  in  Figure  6  to  explore  this.  The  standardized  residuals  of  regressing  the  logit  utilization,  logit 2 fit), 
on  the  predictor  variables  except  log2(c),  arc  graphed  against  the  standardized  residuals  from  regressing 
log2(c)  on  the  same  variables.  The  partial  regressions  are  fitted  using  the  final  hi  square  weights  from  the 
full  model  fit,  and  the  standardization  is  a  division  of  the  residuals  by  the  estimates  mgj  (ip).  Figure  6  shows 
that  log2(c)  has  explanatory  power  for  each  link  separately  and  not  just  across  links.  We  can  also  see  from 
the  plot  that  there  is  a  remaining  small  link  effect,  but  a  minor  one.  This  is  also  demonstrated  in  Figure  7,  the 
same  plot  as  in  Figure  5,  but  for  the  enlarged  model;  the  horizontal  scales  on  the  two  plots  have  been  made 
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the  same  to  facilitate  comparison.  The  major  link  effect  is  now  no  longer  present  in  the  enlarged  model.  The 
result  is  the  same  for  the  same  visual  display  for  the  synthetic  data. 


11.7.8  Alternative  Forms  of  the  Best-Effort  Bandwidth  Formula 

The  Best-Effort  Delay  Formula  of  the  Best-Effort  Delay  Model  in  Equation  5  is 

logit  2(uijk)  =  o  +  oc  log2(cj)  +  oTSlog2(Tidj)  +  ow(-log2(-log2(u;fe))).  (6) 

Since  r  =  075,  the  formula  can  be  rewritten  as 

logit2(n)  =  o  +  (oc  +  orS)  log 2(c)  +  oTs  log2(7 bS)  +  ow(-  log2 (  log2(w))).  (7) 

In  this  form  we  see  the  action  of  the  amount  of  multiplexing  of  connections  as  measured  by  c,  and  the  end- 
to-end  connection  speed  as  measured  by  7*,.  An  increase  in  either  results  in  an  increase  in  the  utilization  of 
a  link. 


11.7.9  Modeling  the  Error  Distribution 

As  we  have  discussed,  our  study  of  the  residuals  from  the  fit  of  the  Best-Effort  Delay  Model  showed  that 
the  scale  of  the  residual  error  distribution  increases  with  the  delay.  The  study  also  showed  that  log2 (mg)  is 
linearly  related  to  log2(<5).  From  the  least  squares  estimates  for  the  live  data,  the  estimate  of  the  intercept  of 
the  regression  line  is  —0.481,  the  estimate  of  the  linear  coefficient  of  the  line  is  0.166,  and  the  estimate  of 
the  standard  error  is  0.189.  (Results  arc  similar  for  the  synthetic  data.) 

We  also  found  that  when  we  normalized  the  residuals  by  the  estimates  rhsj  (7).  the  resulting  distribution  of 
values  is  very  well  approximated  by  a  constant  times  a  f-distribution  with  15  degrees  of  freedom.  Because 
the  normalized  residuals  have  a  median  absolute  deviation  of  1  and  the  1 15  has  a  median  absolute  deviation 
of  0.691,  the  constant  is  0.691  We  will  use  this  modeling  of  the  error  distribution  for  the  bandwidth 
prediction  in  Section  11.8. 


11.8  Bandwidth  Estimation 

The  Best-Effort  Delay  Model  in  Equation  5  can  be  used  to  estimate  the  bandwidth  required  to  meet  QoS 
criteria  on  delay  for  best-effort  Internet  traffic.  We  describe  here  a  conservative  procedure  in  the  sense  that 
the  estimated  bandwidth  will  be  unlikely  to  be  too  small.  In  doing  this  we  will  use  the  coefficient  estimates 
from  the  live  delay  data. 

First,  we  estimate  the  expected  logit  utilization,  £  =  logit2  (it)  by 

£  =  —8.933  +  0.420  log2(c)  +  0.444  log2(rJ)  +  0.893(—  log2(—  log2(u;))). 

On  the  utilization  scale  this  is 

2* 

u  = 

1  +  2^ 

Next  we  compute  a  predicted  median  absolute  deviation  from  the  above  linear  regression: 

m5(V0  =  2-°-481+0-1661°g^. 
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Median  Residual  Logit  Utilization  (logit  base  2  fraction) 


Figure  7:  Dotplot  of  link  median  residuals  for  the  Best-Effort  Delay  Model. 


Let  tis(p)  be  the  quantile  of  probability  p  of  a  t  distribution  with  15  degrees  of  freedom.  Then  the  lower 
limit  of  a  100(1  —  p)%  tolerance  interval  for  l  is 

l(jp)  =  £-  rhs(ip)ti5(p) / 0.691. 

For  p  =  0.05,  ti5(p)  =  1.75,  so  the  lower  95%  limit  is 

1(0.05)  =  i-  2.53m5(V>). 


The  lower  95%  limit  on  the  utilization  scale  is 


£(0.05) 


2^0.05) 

1  +  2^0.05)  ■ 


This  process  is  illustrated  in  Figure  8.  For  the  figure,  7 &  was  taken  to  be  214  bits/sec/connection.  On  each 
panel,  the  values  of  r  and  uj  arc  fixed  to  the  values  shown  in  the  strip  labels  at  the  tops  of  the  panels,  and  u 
and  £(0.05)  arc  both  graphed  against  log2(<5)  for  S  varying  from  0.001  sec  to  0.1  sec. 


11.9  Other  Work  on  Bandwidth  Estimation  and  Comparison  with  the  Results  Here 

Bandwidth  estimation  has  received  much  attention  in  the  literature.  The  work  focuses  on  queueing  because 
the  issue  driving  estimation  is  queueing.  Some  work  is  fundamentally  empirical  in  nature  in  that  it  uses  live 
streams  as  inputs  to  queueing  simulations  or  synthetic  streams  from  models  that  have  been  built  with  live 
streams,  although  theory  can  be  invoked  as  well.  Other  work  is  fundamentally  theoretical  in  nature  in  that 
the  goal  is  to  derive  mathematically  properties  of  queues,  although  live  data  arc  sometimes  used  to  provide 
values  of  parameters  so  that  numerical  results  can  be  calculated.  Most  of  this  work  uses  derivations  of  the 
delay  exceedance  probability  as  a  function  of  an  input  source  to  derive  the  required  bandwidth  for  a  given 
QoS  requirement.  The  delay  exceedance  probability  is  equivalent  to  the  our  delay  probability,  where  the 
buffer  size  is  related  to  the  delay  by  a  simple  multiplication  of  the  link  bandwidth.  Since  exact  calculations 
of  the  delay  probability  arc  only  feasible  in  special  cases,  these  methods  seek  an  approximate  analysis,  for 
example,  using  asymptotic  methods,  stochastic  bounds,  or  in  some  cases,  simulations.  There  has  been  by 
far  much  more  theoretical  work  than  empirical. 
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Log  Delay  (log  base  10  ms) 


Figure  8:  The  QoS  utilization  from  the  Best- Effort  Delay  Model  is  graphed  against  log  delay  given  the 
bit  rate  and  the  delay  probability.  The  bit  connection  rate  7 b  is  taken  to  be  2 14  bits/sec/connection.  The 
upper  curve  on  each  panel  estimates  the  expected  values  and  the  lower  curve  is  the  minimum  values  of  95% 
tolerance  intervals. 


The  attention  to  the  statistical  properties  of  the  traffic  stream,  which  have  an  immense  impact  on  the  queue¬ 
ing,  receives  attention  to  varying  degrees.  Those  who  carry  out  empirical  studies  with  live  streams  do  so 
as  a  guarantee  of  recreating  the  properties.  Those  who  carry  out  studies  with  synthetic  traffic  from  models 
must  argue  for  the  validity  of  the  models.  Much  of  the  theoretical  work  takes  the  form  of  assuming  certain 
stream  properties  and  then,  deriving  the  consequences,  so  the  problem  is  solved  for  any  traffic  that  might 
have  these  properties.  Sometime,  though,  the  problem  is  minimized  by  deriving  asymptotic  results  under 
general  conditions. 


11.9.1  Empirical  Study 

Our  study  here  falls  in  the  empirical  category  but  with  substantial  guidance  from  theory.  To  estimate  ex¬ 
ceedance  probabilities,  we  run  simulations  of  an  infinite  buffer,  FIFO  queue  with  fixed  utilization  using  live 
packet  streams  or  synthetic  streams  from  the  FSD  model  as  the  input  source. 

The  tradition  for  using  live  Internet  streams  in  a  queueing  simulation  began  early  in  the  study  of  Internet 
traffic.  In  a  very  important  study  it  was  shown  that  long-range  dependent  traffic  results  in  much  greater 
queue-length  distributions  (Erramilli,  Narayan,  and  Willinger  1996)  than  Poisson  traffic.  This  was  an  im¬ 
portant  result  because  it  showed  that  the  long-range  dependence  of  the  traffic  would  have  a  large  impact  on 
Internet  engineering.  In  other  studies,  queueing  simulations  of  both  live  and  altered  live  traffic  arc  used  to 
study  the  effect  of  dependence  properties  and  multiplexing  on  performance  using  the  average  queue  length 
as  the  performance  metric  (Erramilli,  Narayan,  Neidbardt,  and  Saniee  2000;  Cao,  Cleveland,  Lin,  and  Sun 
2001;  Erramilli,  Narayan,  Neidbardt,  and  Saniee  2000). 

In  (Mandjes  and  Boots  2004),  queueing  simulations  of  multiplexed  on-off  sources  are  used  to  study  the 
influence  of  on  and  off  time  distributions  on  the  shape  of  the  loss  curve  and  on  performance.  To  improve  the 
accuracy  of  the  delay  probabilities  based  on  simulations,  techniques  such  as  importance  sampling  are  also 
considered  (Boots  and  Mandjes  2001). 

One  study  (Fraleigh,  Tobagi,  and  Diot  2003)  first  uses  live  streams,  Internet  backbone  traffic,  to  validate  a 
traffic  model:  an  extension  of  Fractional  Brownian  Motion  (FBM),  a  two-scale  FBM  process.  They  do  not 
model  the  packet  process  but  rather  bit  rates  as  a  continuous  function.  Then  they  derive  approximations  of 
the  delay  exceedance  probability  from  the  model,  which  serves  as  the  basis  for  their  bandwidth  estimation. 
Parameters  in  the  two-scale  FBM  model  that  appeal-  in  the  formula  are  related  to  the  bitrate  r  using  the 
data.  This  is  similar  to  our  process  here  where  we  relate  the  stream  coefficients  to  c  and  r.  They  also 
used  queueing  simulations  to  determine  delay  exceedance  probabilities  as  a  method  of  validation.  We  will 
compare  their  results  and  ours  at  the  end  of  this  section. 


11.9.2  Mathematical  Theory:  Effective  Bandwidth 

A  very  large  number  of  publications  have  been  written  in  an  area  of  bandwidth  estimation  that  is  referred 
to  as  “effective  bandwidth”.  The  effective  bandwidth  of  an  input  source  provides  a  measure  of  its  resource 
usage  for  a  given  QoS  requirement,  which  should  lie  somewhere  between  the  mean  rate  and  peak  rate. 
Let  A(t)  be  the  total  workload  (e.g.,  bytes)  generated  by  a  source  in  the  interval  [0,i],  the  mathematical 
definition  of  the  effective  bandwidth  of  the  source  (Kelly  1996)  is 

a(s,t )  =  ^  logE[eSj4^]  0  <  s,t  <  oo,  (8) 
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for  some  space  parameter  s  and  time  parameter  t.  For  the  purpose  of  bandwidth  estimation,  the  appropriate 
choice  of  parameters  depends  on  the  traffic  characteristics  of  the  source,  the  QoS  requirements,  as  well  as 
the  properties  of  the  traffic  with  which  the  source  is  multiplexed.  In  the  following  we  shall  discuss  the 
effective  bandwidth  approach  to  bandwidth  estimation  based  on  approximating  the  delay  probability  in  the 
asymptotic  regime  of  many  sources. 

Consider  the  delay  exceedance  probability  for  a  First  In  First  Out  (FIFO)  queue  on  a  link  with  constant  bit 
rate.  In  the  asymptotic  regime  of  many  sources,  we  arc  concerned  with  how  the  delay  probability  decays 
as  the  size  of  the  system  increases.  Suppose  there  arc  n  sources  and  the  traffic  generated  by  the  n  sources 
are  identical,  independent  and  stationary.  The  number  of  sources,  n,  grows  large  and  at  the  same  time 
the  resources  such  as  the  link  bandwidth  ft  and  buffer  sizes  scale  proportionally  so  that  the  delay  5  stays 
constant.  Let  ft  =  nfto  for  some  /3q,  and  let  Qn  be  the  queueing  delay.  Under  very  general  conditions,  it 
can  be  shown  that  (Botvich  and  Duffield  1995;  Courcoubetis  and  Weber  1996;  Simonian  and  Guibert  1995; 
Likhanov  and  Mazumdar  1998;  Mandjes  and  Kim  2001) 

lim  — rU1  logP(Qn  >  <5)  =  1(5,  ft0),  (9) 

n— >oo 

where 

1(5,  fto)  =  inf  sup  (s/3q(5  +  t)  —  sta(s,t )) ,  (10) 

t>  0  s 

and  is  sometimes  referred  to  as  the  loss  curve  in  the  literature.  Let  (s*,t*)  be  an  extremizing  pair  in  Equa¬ 
tion  (10),  then  a(s*,t*)  is  the  effective  bandwidth  for  the  single  source  as  defined  in  Equation  (8),  and 
na(s*,t*)  is  the  effective  bandwidth  for  the  n  sources.  For  a  QoS  requirement  of  a  delay  5  and  a  delay 
probability  w,  approximating  the  delay  probability  P (Qn  >  5)  using  exp(—nI(5,/3o))  (Equation  9),  the 
bandwidth  required  for  the  n  sources  can  be  found  by  solving  the  following  equation  for  ft 


This  gives 


s*ft(5  +  t*)  —  s*t*na(s*  ,t*)  =  —  log  w. 

log  w 


P  = 


s*t* 


-nce(s*,t*)  — 


S*(5  +  t *)  '  '  '  S*(5  +  t*)’ 

which  is  the  effective  bandwidth  solution  to  the  bandwidth  estimation  problem.  If  the  delay  5 
the  extremizing  value  of  t*  approaches  to  oo,  and  the  bandwidth  in  Equation  (12)  reduces  to 


(11) 

(12) 

oo, then 


n  lim  a(s*,t*), 

and  we  recover  the  classical  effective  bandwidth  definition  of  a  single  source  limt*_KX)  a(s*,  t*)  for  the 
large  buffer  asymptotic  model  (Elwalid  and  Mitra  1993;  Guerin,  Ahmadi,  and  Naghshineh  1991;  Kesidis, 
Walrand,  and  Chang  1993;  Chang  and  Thomas  1995).  If  the  delav  5  — ►  0,  then  the  extremizing  pair  t*  — >  0 
and  s*t*  — ►  s  for  some  s,  the  bandwidth  in  Equation  (12)  reduces  to 

n  lim  a(5/t*,t*) 

t*—>  oo  S 

and  we  recover  the  effective  bandwidth  definition  limt*_>0O  a(s/t* ,t *)  for  the  buffer-less  model  (Hui  1988). 

As  we  can  see,  the  effective  bandwidth  solution  requires  the  evaluation  of  the  loss  curve  1(5,  ftft)  (Equa¬ 
tion  10).  However,  explicit  form  of  the  loss  curve  is  generally  not  available.  One  approach  is  to  derive  ap¬ 
proximations  of  the  loss  curve  under  buffer  asymptotic  models,  i.e.,  large  buffer  asymptotic  model  (5  —>  oo) 
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or  buffer-less  model  (5  — >  0),  for  some  classes  of  input  source  arrivals.  For  example,  if  the  source  arrival 
process  is  Markovian,  then  for  some  //  >  0  and  u  (Botvich  and  Duffield  1995) 

lim  1(6,  Po)  —  r/5  =  v. 

8 — >oo 

If  the  source  arrival  is  Fractional  Brownian  Motion  with  Hurst  parameter  //.  then  for  some  u  >  0  (Duffield 
1996) 

lim  I(5,p0)/52~2H  =  v. 

8—>co 

For  on-off  fluid  arrival  process,  it  is  shown  that  as  5  — ►  0,  for  some  constants  rj(/3 o)  and  v(Po)  (Mandjes  and 
Kim  2001), 

1(5,  p0)  ~  rj(Po)  +  v(Po)V5  +  0(5), 
and  as  5  —>  oo,  for  some  constant  6(Pq)  (Mandjes  and  Boots  2002), 

I(6,Po)~9(Po)v(6), 

where  v(5)  =  —  logP(Residual  On  period  >  6).  However,  it  is  found  that  bandwidth  estimation  based  on 
buffer  asymptotic  models  suffer  practical  problems.  For  the  large  buffer  asymptotic  model,  the  estimated 
bandwidth  could  be  overly  conservative  or  optimistic  due  to  the  fact  that  it  does  not  take  into  account  the 
statistical  multiplexing  gain  (Choudhury,  Lucantoni,  and  Whitt  1994;  Knightly  and  Shroff  1999).  For  the 
buffer-less  model,  there  is  a  significant  utilization  penalty  in  the  estimated  bandwidth  (Knightly  and  Shroff 
1999)  since  results  indicate  that  there  is  a  significant  gain  even  with  a  small  buffer  (Mandjes  and  Kim  2001). 

Another  approach  proposed  by  (Courcoubetis,  Siris,  and  Stamoulis  1999;  Courcoubetis  and  Siris  2001)  is 
to  numerically  evaluate  the  loss  curve  1(5,  Po).  First,  they  evaluate  the  effective  bandwidth  function  a(s,  t) 
(Equation  8)  empirically  based  on  measurements  of  traffic  byte  counts  in  fixed  size  intervals,  and  then 
they  obtain  the  loss  curve  (Equation  10)  using  numeric  optimizing  procedures  with  respect  to  the  space 
parameter  s  and  the  time  parameter  t.  As  examples,  they  applied  this  approach  to  estimate  bandwidth  where 
the  input  source  is  a  Bellcore  Ethernet  WAN  streams  or  streams  of  incoming  IP  traffic  over  the  University  of 
Crete’s  wide  area  link.  Their  empirical  approach  is  model  free  in  the  sense  that  it  does  not  require  a  traffic 
model  for  the  input  source  and  all  evaluations  arc  based  on  traffic  measurements.  However,  their  approach 
is  computational  intensive,  not  only  because  the  effective  bandwidth  function  a(s,  t )  has  to  be  evaluated 
for  all  time  parameters  t,  but  also  because  the  minimization  with  respect  to  t  is  non-convex  (unlike  the 
maximization  in  the  space  parameter  s )  and  thus  difficult  to  perform  numerically  (Gibbens  and  Teh  1999; 
Kontovasilis,  Wittevrongel,  Bruneel,  Houdt,  and  Blondia  2002). 

In  the  effective  bandwidth  approach,  one  typically  approximates  the  buffer  exceedance  probability  based  on 
its  logarithmic  asymptote.  For  example,  in  the  asymptotic  regime  of  many  sources,  using  Equation  (9),  one 
can  approximate 

P (Qn  >  5)  ~  exp(-nl(5,p0)).  (13) 

An  improved  approximation  can  be  found  by  incorporating  a  pre-factor,  i.e. 

P (Qn  >  5)  «  K(n,6,Po)exp(-nI(6,Po)). 

Using  the  Bahadur-Rao  theorem,  such  approximation  has  been  obtained  for  the  delay  exceedance  probability 
in  the  infinite  buffer  case  as  well  as  the  cell  loss  ratio  in  the  finite  buffer  case  that  has  the  same  logarithmic 
asymptote  but  a  different  pre-factor  (Likhanov  and  Mazumdar  1998). 
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11.9.3  Theory:  Other  Service  Disciplines 


Some  authors  have  investigated  service  disciplines  other  than  FIFO  such  as  general  processor  sharing 
(li  Zhang,  Towsley,  and  Kurose  1994)  and  priority  queue  (Berger  and  Whitt  1998).  Although  TCP  is  the 
most  dominant  protocol  in  today’s  Internet,  we  do  not  consider  the  effect  of  the  TCP  feedback  control  mech¬ 
anism  since  the  link  we  sought  for  estimating  a  bandwidth  is  not  a  bottleneck  link.  To  account  for  the  TCP 
feedback  control,  other  authors  have  studied  characteristics  of  bandwidth  sharing  for  elastic  traffic  and  in¬ 
vestigated  the  bandwidth  estimation  problem  for  such  traffic  (de  Veciana,  Lee,  and  Konstantopoulos  2001; 
Fredj,  Bonald,  Proutiere,  Regnie,  and  Roberts  2001).  Again  other  authors  have  considered  regulated  input 
traffic  such  as  those  from  a  leaky  bucket  (Elwalid,  Mitra,  and  Wentworth  1995;  Presti,  Zhang,  Kurose,  and 
Towsley  1999;  Kesidis  and  Konstantopoulos  2000;  Cheng-Shang  Chang  and  Song  2001). 


11.9.4  Theory:  Direct  Approximations  of  the  Delay  Probability 

Besides  approximating  the  delay  exceedance  probability  using  the  effective  bandwidth  approach,  some  au¬ 
thors  have  considered  direct  approximations  for  some  special  classes  of  input  traffic  models.  For  example, 
for  a  Markov  modulated  fluid  source,  the  delay  probability  can  be  more  accurately  expressed  as  a  single  ex¬ 
ponential  with  a  prefactor  K  determined  from  the  loss  probability  in  a  buffer-less  multiplexer  as  estimated 
by  Chernoff’s  theorem  (Elwalid,  Heyman,  Lakshman,  Mitra,  and  Weiss  1995).  For  an  aggregate  Markov 
modulated  fluid  source,  the  delay  probabilities  can  be  approximated  by  a  sum  of  exponentials  (Shroff  and 
Schwartz  1998).  For  a  Gaussian  process,  a  tight  lower  bound  of  the  delay  probability  can  be  obtained  using 
maximum- variance  based  approaches  (Norros  1994;  Knightly  1997;  Kim  and  Shroff  1998;  Fraleigh,  Tobagi, 
and  Diot  2003).  These  expressions  can  be  used  in  place  of  Equation  (13)  to  derive  the  required  bandwidth  for 
a  QoS  requirement.  Readers  are  referred  to  (Knightly  and  Shroff'  1999)  for  a  nice  overview  and  comparison 
of  these  approaches  as  well  as  the  aforementioned  effective  bandwidth  approach  for  bandwidth  estimation. 


11.9.5  Theory:  Queueing  Distributions 


We  now  discuss  implications  of  our  stream  coefficient  delay  formula  and  Best-Effort  Delay  Formula  and 
its  relation  to  some  previous  work.  The  stream  coefficient  delay  model  in  Equation  3  implies  that  for  each 
stream  i. 


w  ~  P (Qi  >  5)  «  exp 


log  2  •  2  o-u 


1  —  u 


°w  °s_ 
o  °w 


for  stream  coefficient  m  and  regression  coefficients  ow ,  os-  This  suggests  the  tail  distribution  of  queueing 
delay  is  Weibull  with  shape  parameter  oso~l.  The  Weibull  form  is  consistent  with  the  FBM  traffic  model 
(and  also  the  two-scale  FBM  model),  but  there  the  shape  parameter  is  2  —  2 H.  Notice  that  oso~l  from 
our  analysis  is  0.52  for  the  real  data  and  0.42  for  the  synthetic  data  which  is  quite  different  from  the  shape 
parameter  computed  from  2  —  2 H  =  0.18.  If  the  bitrate  per  connection  yy,  is  a  fixed  constant,  the  Best-Effort 
Delay  Formula  in  Equation  5  implies  that  for  some  constant  o' 


w  ~  P {Qi  >  5)  ~  exp 


°c+°TS 


log  2  •  2  Tj 


1  —  u 


If  oc  +  oTs  =  oV)  and  the  traffic  bitrate  t*  is  a  multiple  of  r,  i.e.  n  =  n,j,  then  the  above  approximation  is 
consistent  with  the  effective  bandwidth  result  with  many  sources  asymptotics  (Equation  9).  In  our  empirical 
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analysis,  we  have  found  ors  +  oc  and  ow  to  be  quite  close  with  the  ratio  (oTs  +  oc)o~l  being  0.97  for  real 
data  and  0.85  for  synthetic  data.  One  of  the  reasons  that  this  ratio  is  not  1  is  possibly  due  to  the  fact  that 
(Equation  9)  is  an  asymptotic  formula. 


11.9.6  Comparison  of  The  Results  Presented  Here  with  Other  Work 

The  work  presented  in  this  article  has  resulted  in  a  simple  formula  for  bandwidth  estimation.  At  the  same 
time,  validation  has  been  extensive,  permeating  all  areas  of  the  work.  Validation  is  carried  out  in  two  ways: 
empirically  and  theoretically. 

The  large  number  of  papers  in  the  area  of  effective  bandwidth  and  other  theoretical  work  cited  above  have 
yielded  much  insight.  The  work  has  posited  traffic  stream  models  and  investigated  the  resulting  mathemat¬ 
ical  properties.  However,  for  best-effort  Internet  traffic,  there  has  not  been  extensive  study  to  determine 
whether  some  posited  model  accurately  describes  the  stream  statistical  properties.  Nor  has  there  been  exten¬ 
sive  empirical  work  in  the  form  of  extensive  empirical  queueing  simulations  to  determine  whether  queueing 
results  for  best-effort  traffic  fit  the  theory.  Consequently,  the  simple  Best-Effort  Delay  Formula,  not  readily 
derivable  without  a  hint  of  the  final  results,  was  not  uncovered. 

The  interesting  paper  cited  above  using  the  two-scale  FBM  model  surely  took  great  pains  to  validate  the 
model  (Fraleigh,  Tobagi,  and  Diot  2003).  One  problem  with  this  approach,  modeling  traffic  bit  flow  as  a 
fluid  rather  than  the  packet  process  as  it  appeal's  on  the  link,  is  that  the  Gaussian  assumption  does  not  take 
hold  until  the  level  of  aggregation  is  quite  high.  Consequently,  the  FBM  model  is  not  a  good  approximation 
until  the  traffic  rate  is  50  megabits/sec  and  above.  So  their  bandwidth  estimation  model  is  not  a  validated  one 
below  50  megabits/sec.  By  contrast,  our  Best-Effort  Delay  Model  is  valid  to  as  low  as  1  megabit/sec.  But 
the  ensuing  methods  used  in  (Fraleigh,  Tobagi,  and  Diot  2003)  to  find  the  QoS  bandwidth  require  a  series  of 
approximations  and  a  worst-case  empirical  method  in  the  estimation  of  parameters.  There  appeal's  to  have 
been  little  checking  of  these  approximations.  The  bandwidth  results  appeal'  to  us  to  be  inaccurate,  possibly 
arising  from  some  of  the  approximations.  First,  as  the  bit  rate  increases  up  to  1  gigabit/sec,  the  utilization 
appeal's  to  stabilize  at  values  less  than  1,  substantially  so  in  some  cases.  As  our  theoretical  discusses  of  rate 
gains  and  multiplexing  gains  demonstrates,  the  utilization  must  increases  to  1  as  the  bitrate  increases.  This 
is  the  case  for  our  Best-Effort  Delay  Model.  For  example,  the  utilization  for  a  delay  of  10  ms,  a  probability 
of  0.01,  and  a  bit  rate  of  1  gigabit/sec,  is  90%  from  Figure  8  of  (Fraleigh,  Tobagi,  and  Diot  2003)  but  is 
98%  for  our  model.  In  addition,  the  model  in  (Fraleigh,  Tobagi,  and  Diot  2003)  works  simply  with  the  bit 
rate  rather  than  decomposing  into  the  number  of  active  connections  times  the  bit  rate  per  connection  and 
using  two  variables,  as  is  done  in  the  Best-Effort  Delay  Model  here.  As  we  have  demonstrated,  theoretically 
and  empirically,  the  bit  rate  is  not  sufficient  to  account  for  the  utilization  since  a  fast  network  and  one  with 
a  high  traffic  connection  load  must  be  distinguished. 


11.10  Results  and  Discussion 

11.10.1  Problem  Formulation 

Suppose  the  packet  stream  —  packet  arrival  times  and  sizes  —  arriving  for  transmission  on  an  Internet  link 
is  best-effort  traffic  with  bit  rate  r  bits/sec  and  number  of  simultaneous  active  connections,  c.  Suppose  the 
link  input  buffer  is  large  enough  that  packet  loss  is  negligible.  Our  goal  is  to  estimate  the  quality-of-service 
(QoS)  bandwidth  ft  in  bits/sec,  or  equivalently,  the  QoS  utilization  u  =  t/ ft,  that  satisfies  QoS  criteria  for 
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the  packet  queueing  delay  in  the  link  input  buffer.  The  criteria  are  a  delay,  5  sec,  and  the  probability,  u,  that 
the  delay  for  a  packet  exceeds  5. 

11.10.2  Other  Work  on  the  Problem 

There  is  a  wide  literature  on  the  bandwidth  estimation  problem.  Much  of  it  is  theoretical  —  mathematical 
results  that  derive  properties  of  queueing  systems.  A  smaller  literature  is  empirical  in  nature,  based  on 
simulations  with  packet  stream  inputs  from  measurements  on  live  links  or  from  models  for  traffic.  The 
classical  Erlang  Delay  Formula  provides  a  simple  formula  that  can  be  used  for  estimation  of  traffic  streams 
that  in  theory  have  Poisson  arrivals  and  i.i.d.  exponential  sizes.  Best-effort  traffic  is  much  more  complex,  it 
is  nonlinear  and  long-range  dependent,  and  there  has  been  to  date  no  simple,  validated  formula  for  it. 

11.10.3  Principal  Result:  The  Best-Effort  Delay  Model 

The  principal  result  of  this  paper  is  a  statistical  model  that  provides  a  simple,  validated  formula  for  the 
estimation  of  bandwidth  for  best-effort  traffic  in  the  same  way  that  the  Erlang  Delay  Formula  does  for  the 
Poisson-exponential  case.  The  model  has  been  validated  through  extensive  empirical  study  and  through 
consistency  with  certain  theoretical  properties  of  queueing. 

The  model  consists  of  the  Best-Effort  Delay  Formula  plus  random  variation: 

logit2(¥)  =  o  +  oclog2(c)  +  oTs  log2 (r 5)  +  ow(-  log2(-  log2(w)))  +  ip. 

ip  is  a  random  error  variable  with  mean  0  and  median  absolute  deviation  mg  (ip),  which  depends  on  S;  log2 
is  the  log  base  2;  and  logit2(u)  =  log2(tt/(l  —  u)).  The  distribution  of  0.691  ip /mg (ip)  is  a  f-distribution 
with  15  degrees  of  freedom.  Estimates  of  the  coefficients  of  the  model  are 

6  =  —8.933  dc  =  0.420  dT$  =  0.444  ow  =  0.893. 

ms  (ip)  is  modeled  as  a  function  of  5:  \og2(ms(ip))  is  a  linear  function  of  log2(h)  plus  random  variation. 
The  estimate  of  the  intercept  of  the  line  is  -0.481,  the  estimate  of  the  linear  coefficient  of  the  line  is  0.166, 
and  the  estimate  of  the  standard  error  is  0.189.  The  bit  rate  r  is  equal  to  cjf,  where  75  is  the  connection  bit 
rate  in  bits/sec/connection.  So  the  Best-Effort  Delay  Formula  can  also  be  written  as 

logit 2(u)  =  0+(oc  +  OtS)  log2(c)  +  OrS  log2(7 bS)  +  Ow(-  log2(-  log2(w))). 

In  this  form  we  see  the  action  of  the  amount  of  multiplexing  of  connections  as  measured  by  c,  and  the  end- 
to-end  connection  speed  as  measured  by  75.  An  increase  in  either  results  in  an  increase  in  the  utilization  of 
a  link. 

The  Best-Effort  Delay  Model  is  used  to  estimate  the  bandwidth  required  to  carry  best-effort  traffic  given  5, 
u),  t,  and  c.  The  QoS  logit  utilization  is  estimated  by 

l  =  —8.933  +  0.420  log2(c)  +  .444  log2(r<5)  +  0.893(—  log2(—  log2(w))), 

so  the  QoS  utilization  is  estimated  by 


l  +  2r 
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The  corresponding  estimated  bandwidth  is  t/u.  For  such  an  estimate  there  is  a  50%  chance  of  being  too 
large  and  a  50%  chance  of  being  too  small.  We  might  however,  use  a  more  conservative  estimate  that 
provides  a  much  smaller  chance  of  too  little  bandwidth.  Let 

mgty)  =  2-°-481+0-16610^) 

be  the  estimate  of  m{8).  Let  tirjp)  be  the  lower  100 p%  percentage  point  of  a  ^-distribution  with  15  degrees 
of  freedom,  where  p  is  small,  say  0.05.  Let 

£(p)  =  £  —  m((5)fi5(p)/0.691 


Then 


u(p)  = 


2  t(p) 


1  +  2^(p)  ’ 

is  a  conservative  utilization  estimate,  the  lower  limit  of  a  100p%  tolerance  interval  for  the  QoS  utilization. 
The  corresponding  estimated  bandwidth  is  r/u{p). 


11.10.4  Methods 

The  Best-Effort  Delay  Model  was  built,  in  paid,  from  queueing  theory.  Certain  predictor  variables  were 
suggested  by  the  Erlang  Delay  Formula.  Theory  prescribes  certain  behavior  as  r,  c,  or  75  increase,  resulting 
in  rate  gains,  multiplexing  gains,  or  fast-forward  invariance,  and  the  model  was  constructed  to  reproduce  the 
behavior. 

The  Best-Effort  Delay  Model  was  built,  in  paid,  from  results  of  queueing  simulations  with  traffic  stream 
inputs  of  two  types,  live  and  synthetic.  The  live  streams  arc  measurements  of  packet  arrivals  and  sizes  for 
349  intervals,  90  sec  or  5  min  in  duration,  from  6  Internet  links.  The  synthetic  streams  arc  arrivals  and 
sizes  generated  by  recently  developed  FSD  time  series  models  for  the  arrivals  and  sizes  of  best-effort  traffic. 
Each  of  the  live  streams  was  fitted  by  two  FSD  models,  one  for  the  inter-arrivals  and  one  for  the  sizes,  and 
a  synthetic  stream  of  5  min  was  generated  by  the  models.  The  generated  inter-arrivals  arc  independent  of 
the  generated  sizes,  which  is  what  we  found  in  the  live  data.  The  result  is  349  synthetic  streams  that  match 
the  statistical  properties  collectively  of  the  live  streams.  For  each  live  or  synthetic  stream,  we  carried  out 
25  runs,  each  with  a  number  of  simulations.  For  each  run  we  picked  a  delay  5  and  a  delay  probability  w, 
simulations  were  carried  out  to  find  the  QoS  bandwidth  (3,  the  bandwidth  that  results  in  delay  probability 
1 0  for  5.  This  also  yields  a  QoS  utilization  u  =  r//3.  We  used  5  delays  —  0.001  sec,  0.005  sec,  0.010 
sec,  0.050  sec,  0.100  sec  —  and  5  delay  probabilities  —  0.001,  0.005,  0.01,  0.02,  and  0.05  —  employing 
all  25  combinations  of  the  two  delay  criteria.  The  queueing  simulation  results  in  delay  data:  values  of  five 
variables  —  QoS  utilization  u,  delay  5,  delay  probability  u,  the  mean  number  of  active  connections  of  the 
traffic  c,  and  the  traffic  bit  rate  r.  The  delay  data  were  used  in  the  model  building. 


11.10.5  Validity  and  Applicability 

Extensive  data  exploration  with  visualization  tools,  some  shown  here,  demonstrate  that  the  Best-Effort  Delay 
Model  fits  the  simulation  delay  data.  This,  of  course,  is  necessary  for  the  model  to  be  valid.  In  addition, 
validity  is  supported  by  the  model  reproducing  the  theoretical  queueing  properties  as  just  discussed. 
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The  validity  of  the  Best-Effort  Delay  Model  depends  on  the  validity  of  the  traffic  streams  used  as  inputs 
to  the  queueing  simulation;  that  is,  the  packet  streams  must  reproduce  the  statistical  properties  of  best- 
effort  streams.  Of  course,  the  live  streams  of  the  study  do  so  because  they  arc  best-effort  traffic.  Extensive 
validation  has  shown  that  the  FSD  models,  used  to  generate  the  packet  streams  here,  provide  excellent  fits 
to  best-effort  packet  streams  when  c  is  above  about  64  connections,  which  for  a  link  where  7/,  is  about  214 
bits/sec/connection  means  r  is  above  about  1  megabit/sec.  For  this  reason,  only  traffic  streams  with  r  greater 
than  this  rate  are  used  in  the  study,  and  the  Best-Effort  Delay  Model  is  valid  above  this  rate. 

The  results  arc  only  valid  for  links  with  a  large  enough  buffer  that  the  packet  loss  is  negligible.  We  have 
used  open-loop  study,  which  does  not  provide  for  the  TCP  feedback  that  occurs  when  loss  is  significant. 
This  restriction  also  holds  for  the  other  work  on  bandwidth  estimation  cited  here. 

There  is  also  a  practical  restriction  on  applicability.  We  have  taken  the  range  of  our  study  to  include  traffic 
bit  rates  as  low  as  about  1  megabit/sec.  We  have  done  this  simply  because  we  can  do  so  and  achieve  valid 
results.  But  even  for  the  least  stringent  of  our  delay  criteria,  5  =  0. 1  sec  delay  and  cc  =  0.05  delay  probability, 
the  utilizations  arc  low  for  rates  in  the  range  of  1  megabit/sec  to  5  megabits/sec.  This  might  well  be  judged 
to  be  too  small  a  utilization  to  be  practical;  if  so,  it  might  mean  that  the  what  must  be  sacrificed  is  the 
negligible  packet  loss,  which  means  that  a  QoS  study  at  the  very  low  traffic  bit  rates  needs  to  take  account 
of  TCP  feedback. 

One  outcome  of  the  dependence  of  the  bandwidth  estimation  on  the  traffic  statistics  is  that  our  solution  for 
best-effort  traffic  would  not  apply  to  other  forms  of  Internet  traffic  that  do  not  share  the  best-effort  statistical 
properties.  One  example  is  voice  traffic. 

Finally,  the  Best-Effort  Delay  Model  provides  an  estimation  of  bandwidth  in  isolation  without  considering 
other  network  factors.  A  major  factor  in  network  design  is  link  failures.  Redundancy  needs  to  be  built  into 
the  system.  So  an  estimate  of  bandwidth  from  the  model  for  a  link  based  on  the  normal  link  traffic  may  be 
reduced  to  provide  the  redundancy.  But  the  model  still  plays  a  role  because  the  bandwidth  must  be  chosen 
based  on  link  traffic,  but  now  the  traffic  in  the  event  of  a  failure  elsewhere. 
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